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Chapter 1 

INTRODUCTION 


1.1 BACKGROUND 

I 

j 

1.1.1 Software Reliability Research Goals 

The software reliability research sponsored by NASA-Langley Research 
Center (NASA-LaRC) focuses on the development of a credible method 
for predicting operational reliability — that is, predicting the improbability 
that the system will fail due to residual faults remaining in the software 
[1]. It is these residual faults, which surface infrequently, that cause the 
rare event or extremely improbable failures. As evidenced by the first 
well-publicized Space Shuttle software bug, the failure of the initialization 
logic in J. Garman’s words resulted from a “very small, very improbable, 
very intricate, and a very old mistake” [2]. This bug typifies the rare and 
convoluted combination of events which causes carefully developed software 
to fail. 

Although considering all faults is important in reliability prediction, the 
most probable faults are often eliminated using the software quality assur- 
ance methods such as those described in the new DoD standards STD-2167 
for software development [3], STD-2168 for software quality evaluation [4], 
and in the certification guidelines described in [5] . In systems critical to 
the flight of civil aircraft, safety requirements impose demanding reliabil- 
ity requirements. Accordingly, the System Validation Methods Branch of 
NASA-LaRC has used a value of 10 -9 as the maximum probability of sys- 
tem failure for a ten-hour flight as an informal standard in the search for 
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a credible reliability prediction method for validating critical software [6]. 
To date no known software has been validated to that extent. 


1.1.2 Software Error Experiments 

As Phyllis Nagel wrote in the first report on the repetitive run experi- 
ments for gathering software error data, “Little software reliability growth 
modeling has been based on feedback gained from controlled experiments” 
[7, page 2]. The method of investigation, then, is to conduct a series of 
controlled experiments which provides this feedback. These experiments 
constitute one aspect of the NASA-LaRC software reliability program, viz, 
the collection and analysis of software failure data of laboratory controlled 
quality. 

The Research Triangle Institute (RTI) has participated in this program 
by conducting software error experiments using the computing facilities of 
AIRLAB at NASA-LaRC. RTI has conducted two software error gather- 
ing studies. Both studies were conducted in a controlled environment to 
(i) emulate the production environment of a software engineer developing 
life-critical software and (ii) as much as possible, hold constant the usu- 
ally varying exogenous factors in actual development environments [8,9]. 
This report describes the first of these studies; specifically, a three-version 
implementation of a radar tracking problem. 

1.2 DEFINITION OF TERMS 

The following lists defines the terms which are used throughout this report. 

• APPLICATION TASK - A software module being tested for reliabil- 
ity, previously referred to as an ATi, or Application Task i. 

• DESIGN STAGE - One more than the number of corrections made 
sequentially to the code under test during a replicate. A correction is 
to be interpreted as “the set of all faults fixed at the same time.” 

• DESIGN STATE or VERSION - An instantiation of an implemen- 
tation of the code under test. During the software fault diagnosis- 
correction process, the program fixes result in several design states or 
versions of the code. 


2 



• PROGRAM or IMPLEMENTATION - An independently coded ver- 
sion of the same functional specification (i.e., one of the application 
tasks). 


• REPLICATE, REPLICATION, or REPETITIVE RUN - A set of test 
cases applied to the code under test. (See Section 2.4.1 for further 
explanation.) 

The use of the terms failure, error, fix, and fault in this report are 
consistent with the definitions given below: 

• FAILURE - A program failure occurs when one or more observed 
output value(s) disagree(s) with the correct output value(s). 

• ERROR - The incorrect element(s) of the observed output value(s) 
at the time of failure. 

• FIX - The minimum code change required to correct an error. 

• FAULT - The conceptual flaw in the program which is corrected by 
a fix. 


1.3 SUMMARY 

The software error data compiled and analyzed as a part of the NASA-LaRC 
program of experimentation and documented in this report, were collected 
with the following specific goals in mind: 

• determining if the error rates corresponding to the (sequentially gen- 
erated) design stages of a program follow a log-linear pattern, 

• testing the hypothesis of equal error rates associated with each known 
fault, and 

• providing additional insight into how software fails. 

With respect to the first goal, analysis of software error data yielded 
an independent confirmation of the results of Nagel, et al. [7] in that the 
error rates of design stages were observed to follow a log-linear pattern, as 
described in Section 3.1. 
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The testing of the hypothesis of equal error rates associated with each 
known fault also confirmed the findings of Nagel, et al [7] . This result 
renders suspect the assumption that the program’s failure rate is a constant 
multiple of the number of residual bugs which underlies some of the current 
software reliability growth models [10]. 

In Section 3.4, the identification of interacting faults provides additional 
insight into the software failure process. The authors suspect that less 
reliance on black box modeling of software reliability growth may prove 
useful for improving the predictive validity of models of software reliability 
growth. 


1.4 RELATED RESEARCH REPORTS 

Additional information about the experiment can be found in NASA CR- 
172553 [11]. Additional information about the automated repetitive run 
modeling tool, AUTOSIM, developed for the purpose of this study, can be 
found in NASA-CR 177930 [12]. The related Boeing Computer Services’ 
study is documented in NASA CR-165836 [13] and NASA CR-16481 [7]. 
N-version experimentation with the radar tracking problem can be found 
in [14] and [15,16] . 
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Chapter 2 

EXPERIMENT OVERVIEW 

2.1 THE CODE UNDER TEST 

2.1.1 The Radar Tracking Problem 

The modules from which error data were gathered were independently 
coded programs for a hypothetical radar tracking problem. Slightly dif- 
fering specifications of the problem exist. The first use of the problem was 
in a 1973 TRW study which dealt with the quantitative measurement of 
software reliability and safety [17] . The problem (specification) was used 
in 1979 in the repetitive run modeling study by Nagel et al. [13] that is 
the forerunner of this study. The version of the specification from which 
the modules used in this study were coded is contained in the recent RTI 
contractor report [11] to NASA. A paraphrased version of the specification 
used in this study has since been used by Knight et al. [15,16] in a study 
of coincidental errors in dissimilar, functionally equivalent (i.e., N- Version) 
software. 


2.2 SOFTWARE DEVELOPMENT 

2.2.1 Task Staffing and Management 

The functionally redundant software components developed as a part of this 
study were coded at RTI by programmers (with 2 to 8 years of program- 
ming experience) using a link to the computational facilities in the AIRLAB 
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at NASA’s Langley Research Center. Senior software engineers and soft- 
ware analysts implemented the error detection algorithms and constructed 
the test harness used. English language specifications were provided to 
the programmers to develop the system components. The specification 
provided was written by the senior systems analyst who also coded and ex- 
tensively tested a comparison version to solve the radar tracking problem. 
The comparison version was coded prior to providing the specifications to 
the programmers and therefore served as a prototype used to debug the 
test harness. The programming activity was managed in a conventional 
fashion with the exception that the programmers were instructed not to 
discuss their code with anyone other than their manager or the senior sys- 
tems analyst who was responsible for answering all specification questions. 
The programmers were instructed to optimize the reliability of their code. 

2.2.2 Programmer Selection 

The moderate to advanced skill level programmers were selected by con- 
sidering the criteria reported by Moher and Schneider [18]. A form based 
on this criteria was used to screen applicants and those considered were 
exposed to a series of interviews by the project staff. A competitive salary 
was paid to attract qualified programmers. 

2.2.3 Data Collection 

2. 2.3.1 Secondary Data 

Data were collected both manually and automatically during code develop- 
ment and repair. These data are primarily descriptive of the development 
process. Manual data collection was achieved through the use of project 
notebooks and special forms. An instrumented data collection environment 
[11] was used to automatically collect data on programmer activity. 

When a program fails during testing, the programmer is notified by 
electronic mail that his or her program has failed. The mail message in- 
dicates if an abend occurred or which outputs are in error. If an abend 
occurred, the trace back message was provided. The input case which the 
program failed to execute successfully was also provided. Changes made 
by the programmer to the failed program were annotated in the code using 
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a standard syntax. These changes were also reported on a program change 
report form. 

2. 2. 3. 2 Faults/Fixes Data 

This data compilation augments a previous manual data collection activity 
using the same radar tracking software implementations. The manual data 
collection activity identified 11 faults in one program, 1 fault in a second 
program, and 20 faults in a third program version. Table 2.1 describes 
the faults observed for each program. These faults are defined by the fixes 
required. Note that fixes 3 and 4 of the third program have been identified 
as invalid fixes. These fixes are fixes for perceived faults that did not exist, 
thus reducing the number of valid faults observed from 20 to 18. Fixes 3 and 
4 have been kept in the table merely to keep the fix numbering consistent 
with the raw data files. A more complete documentation of this manual 
data collection activity and the corresponding analyses can be found in 
Dunham, et al. [11]. 

The execution of the 100 automated replications resulted in no observa- 
tions of an error requiring application of fix 7 to one of the independently 
coded modules to correct the error. Since this fix was applied five times 
during 25 earlier replications (See [11, page 56] ), this lack of observation 
prompted the checking of its validity. 

The logical condition bit CMM(7) is set to 1 if the following logical 
condition as stated in the specifications is satisfied: 

At least one of any n consecutive data points lies a distance greater than e\ 
from the line joining the first and last of these points. 

Fix 7 corresponds to handling of degenerate conditions, i.e. when N > P, 
where P is the number of (x,y) coordinates provided to the subroutine 
which tests the logical condition. Figure 2.1 depicts this subroutine, named 
COND7, with and without fix 7 installed. As shown in this figure, fix 7 
changes code in the COND7 subroutine so that the subroutine exits with 
CMM(7)=0 prior to the execution of a DO LOOP if the upper bound on 
the DO LOOP is less than the lower bound. If the upper bound is less than 
the lower bound in FORTRAN77 and fix 7 is not installed, then the DO 
LOOP is not executed. The control flow bypasses the DO LOOP, executes 
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Table 2.1: Faults/Fix Descriptions 


PROGRAM 



ONE 

1 

FUNCTION ANGLEA - overwrote date in common region by making 
alignment* to input variables. 


2 

CMM(5) • wrong data point was discarded due to erroneous index 
specification in a loop. 


3 



4 

CMM(5) • did not specify logic that if M < 1, CMM(S) 
cannot be met. 


5 

FUNCTION RAe) - program abended due to an out-of*bound argument 
when calling the FORTRAN library routine which 
computes the arccosine. 


6 



7 

Function ANGLEA failed to complete FIX DO 1 by not changing all 
variable names. 


8 

Function ANGLEA program abended due to an out*of-bound 
argument when calling the FORTRAN library 
routine which computes the arccosine. 


9 

CMM(l) - used wrong formula to compute the difference between 
2 points. 
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11 


TWO 

1 

Used integer variable instead of boolean variable jj 



when setting the FC. 1 

THREE 

1 



2 

CMM(13) fix for wrong variable N6 thru Nl | 


3 thru 6 



7 thru 15 

CMM(7) thru CMM(15) inappropriate handling of 
computation when the No. of data points is small. 


16 

CMM(7) - the upper bound of a do loop was incorrectly set. 


17 



18 

■JTTTryng^TTTPfTrrrmr^ir^riilJlIlIirjliiy^iaaMaaiaMi 


19 

CMM(3) - Program returned wrong value from AGLCOS. This fix 
is related to fix 17. 


20 

FUNCTION PERDIS * program abended when all 3 points were 
the same. 
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the statement CMM(7)=0 and returns with CMM(7)=0. Thus, the result 
is the same with and without fix 7 installed. 

Re-execution of the test cases for the manual replications during which 
fix 7 was observed indicated that fix 7 was applied in conjunction with fix 
16 and should not have been applied at all. For these failed cases, fix 16 
corrected the error when applied by itself. Fix 7 did not correct the error 
when applied by itself. 

The above analyses resulted in the determination that fix 7 is an invalid 
fix and should never have been applied. It is similar to fixes 3 and 4 
which were applied to handle the same degenerate condition which was 
observed in other subroutines as a result of errors in the CMM bits. It 
is a fix for a perceived fault that was not there. These invalid fixes were 
the results of a relaxed fault identification procedure used at the start of 
the experiment; a procedure which permitted the programmer to correct 
perceived but nonexistent faults. The procedure was later revised to reduce 
the probability of such erroneous fixes. 

The automated testing did not result in the observation of any new 
faults, and in fact resulted in the consideration of fix 7 as an invalid fix, 
thus further reducing the number of valid faults to 17. Fix 7 has also been 
retained in the table to keep the fix numbering consistent with the raw data 
files. 


2.3 ERROR DETECTION METHOD 

The independently coded modules were run for over 13 million input cases 
in the test harness which relied on the technique of N- Version Programming 
to detect program errors. Approximately 1 million of the cases generated 
the error data that appear in the appendix to this report; the other 12 
million rases were special, extra cases run to investigate the fault interaction 
phenomenon described in Section 3.4 . The test harness is described in an 
earlier contractor report for this study [11] which also contains an appendix 
with error data generated from an earlier set of 2 million input cases. 

N- Version programming involves a voting procedure on the outputs of 
N software modules independent coded to a common specification and op- 
erating upon the same input values [19] Intermediate and final program 
outputs were compared, rather than voted, in this study. Whenever an 
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Figure 2.1: Subroutine C0ND7 


(WITHOUT FIX 7 INSTALLED) 

C AT LEAST ONE OF ANY N CONSECUTIVE 
C DATA POINTS LIES A DISTANCE THAN 
C EPS1 FROM THE LINE JOINING 
C THE FIRST AND LAST OF THESE POINTS 
IMPLICIT NONE 
INTEGER** NLIM.J.K.LOLIM.I 
REAL** PERDIS.DIST 
INCLUDE 'LICCOM.FOR’ 

C NOW INITIALIZE FOR A LOOP 
NLIM = NBIG 
CMM(7)=I 
LOLIM = P-NLIM+l 
DO Isl.LOLIM 
J = I + NBIG-1 
K = I 

IF(DIST(I,J).GT.O.O)THEN 
DO WHILE(K.LT.J) 

K = K + 1 

IF(PERDIS(I,J,K).GT.EPS1)RETURN 
END DO 

C 

C WHEN THE FIRST AND LAST OF 
C N CONSECUTIVE DATA POINTS 
C ARE IDENTICAL THE CALCULATED 
C DISTANCE TO COMPARE WILL BE 
C THE DISTANCE FROM THE COINCIDENT 
C POINT TO ALL OTHERS OF 
C THE N CONSECUTIVE POINTS 
ELSE 

DO WIIILE(K.LT.J) 

K = K-f J 

IF(DIST(I,K).GT.EPS1)RETURN 
END DO 
END IF 
END DO 
CMM(7)=0 
RETURN 
END 


(WITH FIX 7 INSTALLED) 

C AT LEAST ONE OF ANY N CONSECUTIVE 
C DATA POINTS LIES A DISTANCE THAN 
C EPS! FROM THE LINE JOINING 
C THE FIRST AND LAST OF THESE POINTS 
IMPLICIT NONE 
INTEGER** NLIM, J,K,LOLIM,I 
REAL** PERDIS.DIST 
INCLUDE 'LICCOM.FOR’ 

C NOW INITIALIZE FOR A LOOP 
NLIM-NBIG 

C** FIX 7 change* next line from CMM(7)=1 
CMM(7)=0 
LOLIM=P-NLIM+l 
C mm FIX 7 the next two lines 

IF (LOLIM.LT.l)RETURN 
CMM(7)=1 
DO Isl.LOLIM 
J=I*f NBIG-1 
K=I 

IF(DIST(I,J).GT.O.O)THEN 
DO WHILE(K.LT.J) 

K=K + 1 

IF(PERDIS(I,J,K)-GT.EPS1)RETURN 
END DO 

C 

C WHEN THE FIRST AND LAST OF 
C N CONSECUTIVE DATA POINTS 
C ARE IDENTICAL THE CALCULATED 
C DISTANCE TO COMPARE WILL BE 
C THE DISTANCE FROM THE COINCIDENT 
C POINT TO ALL OTHERS OF 
C THE N CONSECUTIVE POINTS 
ELSE 

DO WHILE(K.LT.J) 

K=K-f 1 

IF(DIST(I,K).GT.EPSl )RETURN 
END DO 
END IF 
END DO 
CMM(7)=0 
RETURN 
END 
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output inequality occurred, the testing was halted and the faulty mod- 
ule^) identified, analyzed, and corrected. This test method detected errors 
except when all three modules and the extensively tested version failed 
identically in the same output bit(s). 

In addition to providing software error data for computing component 
version reliability, the N- version test harness provided data on the effective- 
ness of different strategies for selecting an answer from occasionally differing 
outputs. Analysis of this data is reported elsewhere [14]. 

2.4 THE REPETITIVE RUN TECHNIQUE 

2.4.1 Replicates 

A repetitive run technique [13] is used for error rate estimation. This 
approach provides better estimates of the program error rates as well as 
estimates of the error rates associated with the individual faults. It involves 
repetitively testing a software module from its pre-release version through 
the detection and correction of m faults. The testing uses inputs generated 
at random according to a pre-specified program usage distribution. During 
the testing called the first replicate, the faults axe identified and removed, 
and the fixes corresponding to each fault axe saved. Next, the software is 
returned to its initial state and executed with a different set of randomly 
generated inputs. As the errors due to a specific fault axe again detected, the 
corresponding fix is applied and the number of input cases to observation 
of each output error is again recorded. This process is called the second 
replication. By generating additional replicates (i.e. the repetitive run 
technique) an estimate of the error rate can be determined by the program 
design stage, by the specific fault, or by the program design state or version. 

2.4.2 Number and Length 

To determine the number of automated replications to be conducted, the 
number of failures required to accurately estimate p, the probability that 
the program will fail due to a specific fault on a given execution is deter- 
mined using the same argument given in [11]. This determination is based 
on controlling the relative error, r, in the estimated failure probability, p for 
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Table 2.2: Upper Limits for Replication Sample Sizes 


1 — a 

r 

.5 

.25 

.1 

.01 

.901057 

11 

44 

272 

27,200 

.950004 

16 

62 

384 

38,400 

.980194 

22 

87 

543 

54,300 

.998626 

41 

164 

1,024 

102,400 

.999855 

58 

232 

1,444 

144,400 

.999993 

81 

324 

2,025 

202,500 

.999999 

100 

400 

2,500 

250,000 


the allowable risk (1— a) close to 1. That is we wish to determine k, the num- 
ber of replicated observations required such that Pr(|p— p\ > rp) < (1 — a). 
Table 2.2 shows the upper limits for the number of replicated observations 
required for different values of (1 — a) and r assuming p is sufficiently close 
to 0. Based on this table, we chose 100 replications for estimating p. 

The length of a replication was set to 10,000 test cases which is the 
same stopping rule selected for the manual data collection activity. 

The error data collected are in Tables 2 through 32 in the Appendix. 

2.5 THE AUTOSIM TOOL 

Figure 2.2 portrays AUTOSIM [12], the automated error diagnosis and cor- 
rection tool developed to expedite the software error data collection process 
under the repetitive run technique. This tool replaced a programmer with 
one year of experience who was performing the time consuming and error 
prone repetitive run testing task. The figure shows the quasi-static data 
structures which remain relatively constant during testing and the dynamic 
data structures which are updated by either the AUTOSIM software or the 
N- VERSION CONTROLLER software. 

The contents of the quasi-static data structures depend on the code 
under test and are updated only when a new fault is identified. The over- 
write, abend, and output error maps contain information on which code 
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fixes axe associated with different types of faults. The code library contains 
the version of the code after acceptance testing and the code fixes. 

The dynamic structures include a trace describing which faults have 
been diagnosed and corrected during each replication. The system state 
includes the corrected versions of the code, the current replication number, 
the test case number, the input and output for the current test case, and 
synchronization information. 
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Quasi-static Data Structure 


I I 


o 


Dynamic Data Structure 


Process 


Synchronization link 


Figure 2.2: The AUTOSIM Tool 
14 
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Chapter 3 

ERROR ANALYSIS 


3.1 ERROR RATES OF DESIGN STAGES 

The (absolute value of the natural logarithm of the) maximum likelihood 
estimate of a design stage’s error rate is tabulated in Table 3.1 . The 
estimate includes the effect of censored samples and is, of course, based on 
the assumption that a design stage of a software module has a constant 
probability of error per input case. The statistic, for programs 1 and 3, 
is plotted in Figures 3.1 and 3.2 . Also plotted are the natural logarithms 
of the corresponding minimum and maximum times to error of the design 
stages. The plots corroborate the observations of log linear trends that 
were made in the Boeing study [7,13] . The raw error data are in Tables 2 
through 8 in the Appendix. 
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Table 3.1: Error Rates by Design Stage 


PROGRAM 

m 

H 

m 

|inem(e | 

1 n e (M/N(r #> )) 

ln e (MAX(Tij)) 

ONE 

i 

100 






2 

100 

915 

2.21 

■ 



3 

100 

2,171 

3.08 




4 

100 

4,200 

3.74 




5 

100 

9,274 

4.53 

TV 



6 

99 

42,458 

6.06 




7 

91 

185,929 

7.62 

1.39 

■ 


8 

45 

439,878 

. . Ifl 

4.45 



9 

3 

288,225 


7.29 


TWO 

1 

100 

139 

0.33 

0.00 

1.39 

THREE 

1 

WBM 

123 

0.21 

0.00 

1.59 


2 


1,418 

2.65 

0.00 

4.25 


3 

isa 

2,590 

3.25 

0.00 

5.06 


4 


25,073 

5.52 

0.00 

8.06 


5 

■El 

76,918 

6.65 

0.69 

8.85 


6 

■29 

288,440 

8.30 

2.40 

9.20 


7 


386,439 

9.57 

4.62 

9.20 


8 

■9 

182,417 

10.50 

7.13 

9.13 


where: 


i is the index of replications 
j is the index of design stages 

kj is the number of replicates containing a j th design stage in which an error was 
observed by the time of the stopping case of the replicate 

t,j is the time (i.e., number of cases) to observation of an error of the j th design 
stage during the i th replicate or the time for the j th design stage to reach 
the stopping case of the i th replicate - whichever occurred first. (Note that 
r,y is measured from the start of the j th design stage during the i th replicate, 
not from the start of the i th replicate; thus, T;y = 0 for replicates that end 
before a j th design stage is created.) 

mle A j is the maximum likelihood estimate of the error rate associated with the 
j th design stage and is given by mle Ay = | ln e (l - kj/ £*2$ r,y)J. 
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3.2 ERROR RATES OF INDIVIDUAL FAULTS 


On the assumption that individual faults give rise to independent error 
processes, the hypothesis of equal error rates for the individual faults was 
tested by use of the maximum likelihood ratio test described on pages 236- 
239 in Cox and Lewis [20] . However, the expression for the test statistic 
has been modified since the form derived in Cox and Lewis does not account 
for censored data (and granulated time). 

The modified test statistic is 

H = 2'tlk i ln,(k i /K)-» i ln.(s j /S) 

+ (*,• - kj)ln e ((sj - kj)/(S - S'))) 

for 

R 

s i = ^2 tij » 

t=u 

K = 

i = i 


s = 52 sj. 

i - 1 

where: 

i is the index of replications, 

R is the total number of replications, 

j is the index of perceived faults (or, more precisely, fixes), 

J is the total number of uniquely identified fixes, 

tij is the time (counted from the start of replication i) of the first error 
ascribed to perceived fault j (or uniquely identified fix j ) during the 
i th replication or the ending time of the replication if no error was 
ascribed to perceived fault j during the i th replication, 
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kj is the number of replications that contained an error ascribed to per- 
ceived fault j 

The test statistic has, asymptotically, a Chi-squared distribution with the 
degrees-of-freedom parameter equal to one less than the number of uniquely 
identified faults (or more correctly, fixes) considered; for the full error data 
summarized in Table 3.2 , the degrees-of-freedom parameter is equal to J- 
1. For the data in Table 3.2 , using the full data the test statistic equals 
approximately 6975 and 7782 for programs 1 and 3, respectively; for the 
partial data (that exclude from consideration faults for which fewer than 
ten errors were observed) it equals approximately 3630 and 5220. 

Clearly, the null hypothesis (of equal error rates for the individual faults) 
is rejected for both programs at an extremely high level of significance. If 
only the uncensored data are used, the null hypothesis is still rejected at 
an extremely high level of significance for both programs. (Program 2 was 
not considered, since only one fault was ever discovered in the program.) 
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Table 3.2: Summarized Error Data for MLE Ratio Test 


PROGRAM 

Fix 

Number (j) 


100 

E* 

1*1 

ONE 

1 

100 

109 


2 

100 

184 


3 

100 

3,310 


4 

100 

38,585 


5 

100 

6,463 


6 

100 

10,842 


7 

100 

20,555 


8 

95 

302,918 


9 

9 

957,237 


10 

1 

994,072 


12 

2 

990,641 

PARTIAL 

— 


382,966 ” 

FULL 

— 

807 

3,324,920 

THREE 

1 

100 

126 


2 

78 

221,559 


5 

100 

7,964 


6 

100 

4,698 


8 

100 

7,964 


9 

100 

4,698 


10 

100 

4,698 


11 

100 

4,698 


12 

100 

1,988 


13 

100 

7,964 


14 

100 

4,698 


15 

100 

4,696 


16 

100 

5,053 


17 

96 

250,239 


18 

100 

106,885 


19 

5 

978,071 

PARTIAL 

— 

1,474 

637,930“ 

FULL 

— 

1,479 

1,553,000 


where: 


i is the index of replications, 

j is the index of perceived faults (or, more precisely, fixes), 

tij is the time (counted from the start of replication i) of the first error ascribed 
to perceived fault j (or uniquely identified fix j) during the i th replication 
or the ending time of the replication if no error was ascribed to perceived 
fault j during the i th replication, 

kj is the number of replications that contained an error ascribed to perceived fault 

j ■ 
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3.3 DESIGN STATES 


The 100 automated replications of testing resulted in the observation of 
45 versions out of a possible 4,095 versions for the first program, and 36 
versions out of a possible 131,071 versions for the third program as shown 
in Table 3.3. Each of these unique versions constitutes a design state. 
These small numbers of observed versions suggest that a (statistical) order 
of precedence of fault detection and removal exists among all faults. 

Table 3.3 also gives the number of patterns of errors observed and the 
16-bit output vector. These data indicate that (i) a version of the program 
can produce several error patterns (e.g., in the extreme case for the third 
program, one version produced 14 error patterns) and (ii) one error pattern 
can be produced by several distinct versions (e.g., in the extreme case for 
the first program, 20 versions produced the same pattern of errors). The 
latter indicates the unsurprising result that different faults can provoke the 
same error manifestation. 

Tables 9 through 32 in the appendix contain the times to failure of each 
version or design state for the first and third programs respectively. These 
data are summarized in Tables 3.4 and 3.5 which show the combinations of 
faults present and the average life length of each of these design states. 


Table 3.3: Version Statistics 


PROGRAM 

NUMBER OF 
FAILED 
VERSIONS 

NUMBER OF 
SUCCESSFUL 
VERSIONS 

NUMBER OF 
ERROR 
PATTERNS 

MAX. NUMBER 6F " 

ERROR 
PATTERNS 
OBSERVED 
FOR A 
SINGLE 
VERSiq* 

MAX. NUMBER OT" 
VERSIONS 
FAILING 
WITH 
THE SAME 
ERROR 
PATTERN 

One 

43 

2 

33 

13 

20 

Three 

34 

2 

38 

14 

14 


22 




ORIGINAL PAGE 13 

OE ROOR QUALITY. 


Table 3.4: Program One Design State Failure Times 



— FaTTETTS — 
PRESENT 

mmmjm 


— avEragE — 

LIFE LENGTH 

i 

1-12 

105^ 

100 

1.03 

2 

2-12 

113 

39 

\——— 5j5o“ 

3 

3-12 

1,501” 

90 

r - 16.68 

4 

mmnirwm 

225 


HBBBBEISEfl' 

5 

mkexsem 

27o"“ 

B KB ■ 

■■BflflBQSSBI 

6 

■OBESa 

751 


62.58 ] 

7 

5, 8-12 

139 

■BBBBBMM 

■bbbukh^b 

8 

8-12 

BB2X2Q£B 

89 

2768.76 

9 

4-12 

1,492 

ii 

34. ?0 

10 

4, 5, 7-12 

1,124 

25 

44.96 

11 

5, 7-12 

308 

■bbbebmimi 

UflHBBl^LbB 

12 

4, 6-12 

r 2,489 

40 

62.22 

13 

6-12 

1,283 

11 

116.64 

14 

6, 8-12 

611 

6 

101.83 

15 

4, 6, 8-12 

wmtuzm 

21 

98.14 

16 

4, 8-12 

WKESHEHM 

54 

306.63 

17 

9-12 


BBUHBJBEliSiUii 

BBBB^X£E9£lB 

18 

4, 7-12 

6,214 

47 

132.21 

19 

7-12 

4,859 

30 

~~ 161.97 

20 

1, 3-12 

12 


BBHBBHE&lBI 

21 


373 


21.94 | 

22 

4, 9-12 

4,345 

bub ■ 

BB 1 SUB 

23 

■rinruB 

520 

30 

17.33 

24 


352 

14 

25.14 

25 

MXKXSm 

55 

bmbhejhbbbi 


26 

■KKK2£B 

58 

■1 HJB ■ 

H K£i 

27 

3, 7-12 

69 

Bi^BBUBBBHI 


28 

■ min 

6,320 

Bi BEBI BB 


29 

■ ii irum 

248 

■ BU ■ 


30 

4, 5, 9-12 

166 

| u 


31 

5, 9-12 

38 

i 

38.00 

32 

5-12 

65 

3 

21.67 

33 

6, 10-12 

8,712 

2 

4,356.00 

34 


9 

B flfl B 

BBBMBBKXiHB 

35 

■BBBBI 

13 

l 

13.00 

36 

3, 8-12 

57 

. * _ _ 

57.00 

37 

3, 5-12 

20 

2 

10.00 

38 

2, 4-12 

7 

B Bfl 1 

BflHHBBQQB 

39 

3, 6-12 

49 

HU BI 

BBQLiIlB 

40 

■BSSOI 

2 

bb bi 

BEdB 

41 

!■ ifiiirm 

I u 

■JB ■ 

BHBBHKSm 

42 

BBBBBISEfl 

33,764 

i 

33,764.00 

43 

9, 11, 12 

5,930 

l 

5,930.00 

44 

9, 10, 12 

2,738 

BUB IB 

BBBBBliUali&laB 

45 

8, 10, 12 

306 

i 

HBBHBE3SSCSB 
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Table 3.5: Program Three Design State Failure Times 


■BuEu3 

FAULTS 

PRESENT 

UFt 

LENGTH 



OBSERVATIONS 

AVERAGE 

LIFE LENGTH 



100 

123 

1.23 

2 

2, ft, 6, 8-20 

89 

l,3ft9 

18.27 

3 

— — H TfTiTB 

8 

58 

7.2ft 

4 

ft, 6, 8-11, 13-20 

37 

929 

25.10 

8 

16-20 

23 

>84 

34.09 

ft 

17-20 

69 

47,830 


7 

17, 19-20 

47 

133,183 


8 

ft, 8, 13, 16-20 

20 

484 

24.20 

9 

ft, 8, 13, 17-20 

33 

2210 

86. 9 > 

10 

r"~“~ 2, 16-20 

18 

1,167 

64.83 

11 

2, 17-20 

21 

17,18ft 

818.33 

12 

2, 18-20 

ft 

4,081 

816.20 

13 

2, 5, 6, 8-17, 19, 20 

1 

6 

6.00 

14 

5, 8, 13, 16, 17, 19, 20 

1 

70 

70.00 

IS 

5, 8, 13, 17, 19, 20 

4 

458 

114.50 

16 

18-20 

22 

28,502 

“ 1, >95.55 

17 

5, 6, 8-11, 13-15, 17-20 

31 

1,506 

48.58 

18 

1 , 2, 5. 6, 8-11, 13-20 

2 

4 

2.00 

19 

MBEiafluimi 

8 

97 

12.12 

20 


17 

42,467 

2,498.06 

21 


19 

353 

18.58 

22 


6 

13ft 

22.80 

23 

— ii !■ iiiii n i ■ 

7 

208 

29.71 

24 

2, 5, 6, 8-16, 18-20 

1 

6 

6.00 

26 

5, 6, 8-11, 13-16, 18-20 

2 

18' 

9.00 

26 

5, 6, 8-11, 13-15, 18-20 

1 

11 


27 

19, 20 

ft 

v ft46,351 



2, 19, 20 

2 

“ 148,825 

74,412.50 

wmzmi 

2, 16, 18-20 

1 

54 

54.00 

■KH 

ft, 8, 13, 18-20 

1 

(“ 100 

r~“ loo.oo 

31 

ft, 8, 13, 16, 18-20 

1 

4 

4.00 

32 

16, 18-20 

2 

103 

51.50 

33 

1, ft, 6, 8-11, 13/20 

- 1 

2 

2.00 

34 


1 

97 

97.00 

35 


1 

5,930 

5,930.00 

36 

20 

1 

16,009 

16,009.00 


24 










3.4 INTERACTING FAULTS 


The data in Table 3.6 were generated by special versions of one of the tested 
programs (specifically, faults 7 and 8 in program 1) operating upon identical 
input to the versions. The first column of the table can be considered to 
contain data from program 1 with only fault 7 present; the second column, 
program 1 with only fault 8 present; the third column, program 1 with 
both 7 and 8 but no others present. “S”s indicate successful operation; 
“F”s indicate failure. Thus, the first row of the table indicates that for 
1,714,177 randomly chosen cases, the three version agreed on the correct 
output (the inputs to the three versions being the same in a case). 

The phenomenon represented by these data has been called “fault in- 
teraction”: to wit, two (or more) faults are said to be interacting faults 
when the error set (the set of points, from the input space of the mod- 
ule that translates into erroneous outputs) that exists when the faults axe 
jointly present in the code differs from the set that is the union (in the 
mathematical sense) of the error sets of the faults separately (or in other 
combinations) present in the software. 

Interacting faults were discovered serendipitously during this experi- 
ment because of a sometimes symptom of interacting faults. The symptom 
is the occurrence of an erroneous output that can be corrected by the repair 
of either of several seemingly unrelated faults - seemingly unrelated in the 
sense that they are logically unrelated from the perspective of their origins 
or causes; obviously they have some relationship in their synergistic effect 
on the computation. In Table 3.6 this corresponds to the S/S/F event that 
occurred 4990 times. A conventional debugging process is likely to miss 
this symptom because, upon detecting an error in the module containing 
both faults, a programmer will most likely correct just one fault (which 
ever one he discovers first) and never know that he had a choice. But the 
repetitive run technique is well suited for observing the option. And be- 
cause of this “either-or” symptom during the generation of the error data 
that is collected in the appendix, several such interacting fault pairs were 
serendipitously discovered - faults 7 and 8 of program 1, faults 2 and 13 
of program 3, and the triplet of faults 7 (later determined to not be a real 
fault), 16, and 20 of program 3. (Fault 20 is not listed in Table 3.2 or in 
the appendix because it did not cause an error and was not detected until 
well after the generation of the data in the appendix was completed and 12 
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million special cases were being run to seek and to examine the interaction 
phenomenon among fault pairs.) 

Table 3.6: Counts for Interacting Faults 


Fault 7 
present 

Fault 8 
present 

Faults 7 & 8 
present 

Number 
of Cases 

S 

S 

S 

1,714,177 

S 

S 

F 

4,990 

s 

F 

S 

349 

s 

F 

F 

19 

F 

S 

S 

473 

F 

S 

F 

0 

F 

F 

S 

1,122 

F 

F 

F 

12 


Consider the following examples. It could happen that the error sets 
for faults jointly present or separately present could be approximately the 
same “size” but consist of different points (clearly the case for the value 
4990 as opposed to the values 349, 473, and 1122 in Table 3.6 ) - so that 
after the detection of an error and proper correction of one of the faults, 
inputs that had previously tested out as not generating errors could be in 
the resulting error set. 

Or it could happen that the error set when two faults are jointly present 
in code is much smaller than the error set of either fault taken separately 
(e.g., if the 4990 had been 10 in Table 3.6 ); in such a case, the faults 
could be considered to be almost compensating or mutually masking - 
so that upon the eventual detection of an error, if only one of the faults 
were corrected (and properly corrected), the error rate of the code would 
increase. 

Although there are insufficient data to support statements about the 
significance of the phenomenon in reliability modeling, it is clear that the 
phenomenon is a mechanism that can give rise to insidious effects that 
plague software testing theory by causing any modification of software to 
leave all previous testing suspect. 
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Chapter 4 

CONCLUDING REMARKS 


The report presents the results of an experiment in software reliability based 
on program samples of a radar tracking problem, N-version programming 
as an error detection mechanism, and automated fault identification and 
correction. 

Testing the software modules with over three million input cases (of 
which two million are reported in the earlier report to this study [11] ) 
corroborated the findings of a previous study [7,13]: the log- linear pattern 
of error rates of design stages and rejection of the hypothesis that all faults 
in a program have the same error rate. 

Additional testing (approximately twelve million input cases) and anal- 
ysis of the resulting error data indicated that there is a fault interaction 
phenomenon that complicates the estimation of the error rates to be associ- 
ated with some faults. The frequency of interacting faults in software and, 
therefore, the importance of accounting for this complicating phenomenon 
in the modeling of software reliability is not yet known. 
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TABLE 1. Seeds Used with the Pseudo-Random Number Generator 


REPLICATION 

SEED 

I 

1050554872 

26 

1765936978 

51 

2008687904 

71 

1348542162 

89 

207784072 


33 
















2: Input Cases to Failure 
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TABLE 2. PROGRAM:ONE, FAULTS:1-12, REPLICATIONS:1-40 
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TABLE 3. PROGRAM:ONE, FAULTS:1-12, REPLICATIONS:41-80 
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TABLE 4. PROGRAMtONE, FAULTS:1-12, REPLICATIONS:81-100 
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3 

9 

■E3B 

66 

75 

mam 

KUUI 

— 

— 

— 

— 

96 

2 

2 

2 


mm 

26 

wmsm 

■HI* 

— 

— 

— 

— 

99 

1 

2 

HI 

408 

10 

36 

mam 

- 

— 

— 

— 

— 

100 

1 

2 


82 

5 


msm 

HIM 

— ■ 

i — 

— 

— 


37 




TABLE 5. PROGRAMrTWO, FAULTS:!, REPLICATIONS:1-100 




ORIGINAL PAGE Ig 

OE POOR QUALITY 


TABLE 6. PROGRAM:THREE, FAULTS:l-20, REPLICATIONS:1-40 


■irem 


■Mm 


MM 

IBM 

Hft 

■Uft 


mu 

mim 

HU 



HUM 


Hum 

mum 

■turn 

EU 1 



mim 

mem 

HEIH 

mu 


■L^ft 

mu 

mm 

UETiM 

mu 

mu 


ftEEIft 

■HU 


a 

■m 

HH 

mm 

mm 


mu 

mu 

mu 


HU 


mu 

mu 

mu 

ftLLflft 

miu 

Ha 

Hal 


— 












HU 

mmm 

KbUft 

■Eft 

■a 


mm 

wmm 

iftift 

m 

m^ft 



mm 

mft 



mu 


mmm 

■□aft 

HD 

C3 

HEm 


mmm 

Mftft 

Hftft 

Hftft 

HMLft 

■Eft 

ma 

■Eft 

Hftft 

imft 

HI 

mu 

ftftCM 

mu 

ma 

ma 

'^M^M 

mm 

m 

Mvzm 

■n 

Mim 

HHft 

imft 

Hftft 

mft 

heu 

Hft 

mu 

^LXft 

mmm 

mu 

mmssm 

BOi 

Ih 

mm 

mm 

m?zm 

Kl 

MVZM 

KrlB 

mu 

mu 

KM 

EEU 

mM 

mu 

mm 

MlXf’M 

■ftUI 

mai 

HI 

imftft 



MU 

MU 

mu 

■Eft 


HHft 

mft 

mu 

HHft 

mu 

mu 

HEU 

EM 

mi 

Eftl 

■EMI 

ra 

mem 

mm 

fftftft 

mu 

mu 

HHft 

RMft 

mft 


HHft 

Hftft 

mu 

E2U 

mum 

ma 

ma 

■Cfti 

■w 

mm 

IHU 

mm 

mu 

mu 

mu 

mu 

mm 

mu 

mift 

mu 

mu 

mum 

mim 

■ms 

ma 

HQM 

mm 


mm 




■uft 

mu 

Hftft 

mu 

mu 

mu 

mu 

■ETTTH 

mmm 

ma 

mai 

HOI 

mm 

IBB 


mu 

mu 

heu 

Hftft 

Hftft 

Mft 

HHft 

HHft 

iHft 

Hftft 

EHU 

ehm 

mi 

o 

hQH 


B5B 

hm 

mu 

HU 

HU 

^EEft 

mu 

HU 

mu 

^KEH 

■Oft 

mu 

ftE2H 

mu 

mss 

mai 

«£■ 

MM 

ft m 

mm 

mu 

ift^ft 

hm 

mu 

mu 

EU 

mu 

mu 

Hft 

^EU 

miu 

miu 

ma 

ma 

BEQH 

!■ 



mu 

HU 

^Hft 

mu 

HHft 

mft 

mu 

imft 

Hftft 

mu 

huu 

miu 

Ha 

mai 

.HCftft 

ks 

■Eft 

MIDI 

mu 

HU 

mu 

mu 

mu 

KM 

mu 

H9 

mu 

mu 

mum 

miu 

mai 

ma 

Imam 

mm 

mm 

Hfclft 

mu 

mu 


mu 

mu 

KM 

mu 

mu 

mu 

mu 

mmm 

■mu 

ma 

KO| 

hh 


warn 

HLU 


mu 

HEM 

mu 

mu 

KM 

mu 

■EEft 

mu 

mu 

mmm 

m eu 

ma 

m 

Immi 

a 

o 

dft 

HSU 



mm 

mwm 

mvm 

TYB 

EXM 

EXM 

■Eft 

huu 

miu 

wmom 

ma| 

■EU1 

mm 


HU 

mu 

mu 

HU 


mu 

HU 

■Oft 

HHft 

mu 

mu 

■mu 

mmm 

miu 

ma 

■EMI 


mft 

HLift 

mu 

mu 

HEU 

■EU 

HU 

Mft 

mu 

mu 

Hlft 

mu 


mmm 

ma 

■SI 

HEEHI 

S 

HOi 

■Hfl 

mu 

mu 

Hftft 

Hftft 

Hftft 

mft 

imft 

mm 

Hftft 

mu 

m&m 

hem 

ma 

ma 

ng 













mu 

HEM 

mu 

mai 

ail 

1km 

a 

■Eft 

mv$m 

MM 

EM 

mu 

heu 

mu 


WtiiM 

mu 

mu 

mu 

mi 

miu 

m3 


Ki 



MQft 




mu 

mu 

HU 

mu 

HU 

mft 

mu 

WffiFM 

■HU 




— 

m 

mem 

mu 

mu 

mu 

mu 

mu 

EM 

mu 

mu 

mu 

mu 

M\UM 

■reft 

HO 


Igft^m 

MM 

mm 

■Lift 

mu 

mu 

mu 

mu 

mu 

KM 

mift 

mu 

mu 

■Efti 

KEBftl 

1 *zw 

H=ft 





mm 

mu 

mu 

mu 

■EU 

mu 

IKE 

mu 

mu 

mu 

mu 

Mwzm 

hem 

ko 


heob 

MM 

mm 


ME U 

HEft 

heu 

mu 

heu 

EU 

mu 

mu 

mu 

mu 


HEM 

ma 


HE3H 

W 

mm 

EXM 

mm 

EXM 

HEU 

heu 

heu 

■£■ 

EXM 

■EU 

heu 

mu 

mmm 

mu 

■■ca 

mai 

mim 

— 




mu 

Hftft 

B 

HEB 

mft 

HHft 

mu 

mu 

mu 

mmu 

heu 

■HBft 

IBM 

HEEM 


Kft 

mm 

■Dft 

HXU 

mu 

■EM 

HIU 

HU 

■EU 

EEB 

heu 

mu 

EHU 

hm 

mma 

ma 

WKE^I 


CM 

mm 


MEM 

mu 

HOft 

mu 

EM 

mu 

mu 

mu 


■ht-m 


MK-IM 

ma| 

HE£^I 

mm 

KIM 

■EE ft 

mu 

■EM 

mu 

mu 

mu 

EU 

heu 

mu 

mu 

HHft 

MIM 


mca 


■n 



ttfl 



mu 

iftm 

mu 

■U 

■Oft 

mu 

ftKJft 


m£u 

miu 

■ma 

mai 

— 

s 



HU 

mu 

mu 

Hftft 

HHft 

mft 

HHft 




Hftlft 


ma 


mm 

mm 

mm 

Utl 

heu 

M&M 

mu 

heu 

HEU 

EU 

■ESI 

■tw 

heu 


■QU 


ms 




mm 




mu 

Hft 

mu 

HU 

HBftl 

mu 

mu 


mftu 


■Hi 

maj 

mui 

mM 

■Eft 


mu 

mg 

mu 

mu 

mu 

mft 

mu 

mu 


mu 

Mvium 

mui 



mi 

MM 



mift 

■’EM 

mu 

■EU 

■EM 

EU 


mu 

mu 

mu 

mu 

ftggft 

■a 
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TABLE 7. PROGRAM:THREE, FAULTS: 1-20, REPLICATIONS:41-80 


REP 



n- 


r-5— 



§ 

MSM 

11 

mm 

msm 

14 

15 

KH 

m- 

“Tl— 

19 


41 

u 

mm 

kb 

BOB 

KB 

63 

—83 

83 

mm 

■JMi 

83 

83 

M 1 M 

■1XB 

wmsm 

— 

— 

42 

i 

mm 

nr 

1^ 

KB 

17 

B KB 

17 

Ui 

kb 

it 

ir 

kb 

mmm 

■JJJi 

— 

— 

43 

i 

7 

KIB 

10 

10 

10 

msm 

10 

n- 

10 

10 

10 

kb 

■UM 


— 

— 

44 

i 

— 

mm 

19 

19 

29 

KB 

19 

BIB 

19 

1$ 

19 

keb 

keib 

K£B 

— 

— 

45 


9 

42 

39 

BOB 

39 

39 

39 

9 

42 

kb 

wmj-m 

64 

helk 

PUB 

— 

— 

46 

mm 

2 

68 

10 

68 

10 

10 

10 

2 

66 

KB 

KB 

f 

kieb 

mmm 

— 

— 

47 

mm 

6 

57 

57 

57 

57 

57 

5* 

6 

si 

f 

57 

keb 

mmm 

Tf 

— 

— 

46 

mm 

mm 

KB 

74 

74 

74 

KB 

74 

BIB 

74 

KB 

74 

77 

mmm 


— 

— 

49 

mm 

mm 

15 

13 

15 

13 

BBB 

msm 

BEB 

15 

13 

KB 

9 

■TOH 

KIB 

— 

— 

50 


mm 

mm 

KB 

96 

96 

98 

msm 

KIB 

98 

98 

-S8 

33 

keb 

KEB 

— 

— 

51 


• 

msm 

54 

msm 

54 

54 

54 

BIB 

MT*m 

■IK 

KB 

KB 

mum 

Km 

— 

— 

52 

mm 

KIM 

«» 

20 

49 

20 

20 

20 

BIB 

mm 

20 

kb 

kb 

msm 

■rpf 

— 

— 

53 

mm 

KIM 

mvxm 

Baa 

mvrm 


msm 

msm 


keb 

keb 

msm 

« 

KIB 

mum 

— 

— 

54 

i 

— 

3- 

3 

3 

3 

3 

3 

3 

3 

3 

3 

keb 

Ksa 

kieb 

— 

— 

55 

mm 

— 

46 

40 

40 

40 

40 

40 

BIB 

40 

40 

40 

■If 


keub 

— 

— 

56 


3 

20 

3 

20 

3 

3 

3 

3 

20 

3 

3 

61 

- 

keb 

— 

— 

57 


1 

msm 

iu 


Him 

TIB 

Kim 

1 

Tf 

KEB 

BEB 

$ 

mmm 

keek 

— 

— 

58 


mm 

« 

62 

KB 

62 

62 

nr- 

kib 

mm 

SS- 

62 

keb 

KUfl 

Km 

— 

— 

59 


mm 

keb 

■ UB 

■if 

gig 

mrxm 

mam 

mm 

bbb 

msm 

■if 

smm 

KIB 

HU 

— 

— 

60 

mm 



HEfl 

HKB 


HEB 

kb 

msm 

■DU 

KEB 

BBB 

59 

mmm 

Km 

— 

— 

61 

mm 

mm 

msm 

mm 

KEB 

29 

29 

29 

■f 

keb 

BUB 

kb 

hub 

mmm 

KIB 

— 

— 

62 


— ~ — 

* 

2 

KB 

2 

2 

2 

2 

5 - 

2 

* 

*0 

KIEB 

Km 

— 

— 

63 J 

i 

3 

BUB 

25 

um 

25 

25 

25 

3 

hub 

KB 

KIB 

KB 

Ml 

■TCf 

— 

— 

flEXflR 

mm 

mum 

43 

16 

43 

16 

16 

16 

BIB 

45- 

16 

HUB 

hub 

K£M 

KIB 

— 

— 

65 



19 

19 

19 

19 

19 

19 

KIB 

■KIB 

KB 

BUB 

HUB 

kieb 

KEB 

— 

— 

66 

i 

mm 

45 

■M 

45 

■nn 

KB 

BOB 

mm 

45 

45 

45 

kb 

Ml 

KEB 

— 

— 

67 



29 

29 

29 

29 


MlSM 

mm 

29 

29 

29 

■IB 

EH 

mmm 

— 

— 

68 

mm 

HB 

T» 

14 

14 

14 

msm 

bub 

mm 

KB 

KB 

14 

99 

—vr 

keb 

— 

— 

69 

mm 

— 

KB 

19 

19 

19 

BUB 

19 

BIB 

19 

19 

19 

18 

kieb 

KIEB 

— 

— 

70 



BOB 

13 

13 

13 

13 

13 

BEB 

13 

msm 

km 

■M 

■TIM 

K2B 

— 

— 

71 

'iMm 

mm 

KB 

16 

21 

16 

KB 

18 

mm 

21 

IB 

kb 

kh 

kieb 

Km 

kieb 

— 

72 

i 

mm 


11 

24 

11 

nr 

11 

BEB 

24 

11 

11 


MUM 

K2B 

— 

— 

73 

mm 

MXM 

TW 

60 

BOB 

60 

0 

60 

■f 

BEB 

KM 

kb 

hub 

K m 

KEB 

— 

— 

74 

mm 

mm 

KEB 

m\FM 

msm 

wnjm 


KM 

mm 

KM 

msm 

msm 

msm 

kieb 

KEB 

— 

— 

75 

mm 

mm 

BOB 

0 

HUB 

bb 


msm 

BIB 

msm 

msm 

msm 

HU 

WMSM 

h nm 

— 

— 

76 

mm 

mm 

BOB 

86 

msm 

66 

86 

66 

Km 

KM 

86 

kb 

kb 

mmm 

m.mm 

— 

— 

77 


— 

69 

69 

69 

69 

69 

KB 

f 

69 

69 

f 

22 

KOI 

keb 

— 

— 

78 

s 

» _ 

KB 

42 

Kl 

42 

KB 

42 

1 

42 

42 

KB 

»-f 

Uil 

kebb 

— 

— 

79 


mm 

bub 

16 

BUI 

16 

BUB 

16 

3 

KM 

16 

16 

3 

KiK 

KEB 

— 

— 

80 


zn 

BOB 

msm 

msm 

mem 

BUB 

msm 

1 

BEB 

BEB 


10 

mm 

■SB 

— 

— 


(jkivaLNAL PAGE 13 

OE EQOR QUAL1TX 
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ORIGINAL PAGE IS 
OP POOR QUALITY 


TABLE 8. PROGRAM:THREE, FAULTS:l-20, REPLICATIONS:81-100 


REP 



i 

2 




9 

10 

11 

HIM 

13 

14 

15 

16 

1 7 

18 

19 

HIM 

81 

i 

8 

be 

56 

KM 

56 

56 

56 

8 

56 

56 

56 

53 

helm 

mtzm 

— 

— 

62 

i 

1 

mivm 

KM 

■TFM 

mtm 

43 

43 

1 

■liM 

43 

43 

7 

mmm 

KIM 

— 

— 

63 

mm 

mm 


35 

^1 

HEM 

35 

35 

KIM 

71 

35 

35 

29 


hem 

M^3i 

— 

64 


mm 

80 

33 

50 

HUM 

33 

33 

mm 

50 

33 

33 

21 

■ray 

199 


— 

65 

i 


49 

49 

49 

49 

49 

49 


49 

49 

49 

12 

hehm 

HEM 

— 

— 

66 

2 

mm 

ggj 

95 

Kk£M 

95 

95 

95 

KM 

■ irfM 

95 

KM 

6 


HEIM 

— 

— 

87 

1 

mm 

m 

96 

HUM 

96 

96 

96 

KIM 

96 

96 

96 

37 


■rvy 

— 

— 

88 

1 


76 

9 

76 

9 

9 

9 

9 

76 

9 

9 

8 

290 

23 

— 

— 

89 

1 

» 

21 

6 

21 

8 

6 

8 

5 

21 

8 

8 

46 


heem 

— 

— 

90 

mm 

mm 

MEM 

1MB 

KM 

KM 

Bil 

■liM 

mm 

HEM 

HEM 

hem 

4 

■my 

mam 

— 

— 

91 




n 



6 

6 

6 

6 

6 

6 

59 

mzvxm 

huim 

— 

— 

92 

t- 

mm 

75 

87 

mtm 

57 

H EM 

57 

1 

75 

57 

57 

23 

■ray 

KI11M 

— 

— 

IHEENI 

mm 


23 

8 

mtm 

8 

8 

6 

8 

mtm 

8 

8 

61 

■my 

■»*<y 

— 

— 



— 

54 

54 

■m 

54 

54 

54 

KIM 

54 

54 

54 

17 

■ray 

mam 

— 

— 

95 

mm 

mm 

mtm 

18 

22 

18 

18 

18 

HEM 

22 

18 

18 

27 

heem 

■w 

— 

— 

96 



BU 

■ IM 

■ »iM 


M¥IM 

hum 

hem 

msm 

HUM 

■ UM 

KM 

him 

19 

— 

— 

97 

i 

Em 

mm 

■ MM 

KM 

hum 

mm 

HUM 

him 

HUM 

HEIM 

HEIM 

HUM 

mum 

581 

— 

— 

98 

i 

mm 


26 


26 

26 

26 

»0 

51 

26 

26 

2 

m\nm 

HUM 

— 

— 

99 

i 



48 

48 

48 

48 

48 

EM 

48 

48 

48 

1 


606 

— 

— 

HEMI 

i 


mvvm 

33 


33 

33 

33 

EM 

hum 

33 

33 

MEM 

■ray 

mtm 

— 

— 
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3: Design State Failure Times 
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'ORIGINAL PA'G 7 ? ** 
POOR QL'ALITY 


TABLE 9. PROGRAMrONE, VERSIONS.1-13, REPLICATIONS: 1-25 


l mMm 

——n i i EflaflaaajHum nm Mi^eeh— 

■Mi 

■ ■ 

B1 

HUH 

u 

u 

blh 

EH 

bhhb 

KE 

Bill 

EUR 

HLH 

KM 


u 

wmm 

HM 

Hi 

EH 

mm 

mm 


bse 

IEH 

ESE 


ESH 

ni 

mm 

hlh 

KLH 

KS 

bsh 

HBSE 

mm 

HELM 

km 

BU 

him 

bse 

EBB 


mm 

B3H 

HI 

bsh 

bei 

bee 

BSI 

HITE 

hlh 

BSH 

bbe 

KM 

KLM 

ni 

mm 

Bsn 

hlh 

H£H 

KLH 

B^E 

bsh 

HUHH 

bsh 

IBE 

hee 

BE 

bee 

HB^H 

mm 

BSI 

hlh 

BSH 

o 

■EE 

^E 

Bk l ->H 

KM 

HH 

bbe 

be 

H^E 

I^HHH 


be 

KH 

bsh 

EH 

ke 

EE 

ksm 

KM 

Bli 

ee 

^SK 

HBE 

nUH 

■n 

bsh 


EH 

be 

bllh 

BSE 

bee 

E 

HEM 

esh 

EE 

KSH 

hqH 

— 

a 

HI 

BSE 

bse 

EM 

be 

kkfh 

eh 

em 

^Bi 

BEI 

HEH 

HHJB 

mm 

B1 

O 

m 

EH 

B^E 

be 

BEDE 

bsh 

ESI 

E9 

EH 

ESH 

9KQH 


BS3B 

mm 

bsh 

EE 

bee 

EE 

TT.il 

BSE 

BE 

B 

Gfl 

BK2H 

HU 

mm 

IKS 

KH 

EH 

EE 

em 

EH 

m mm 

ESH 

HUH 

BE 

IBB 

KESI 

BBE1H 


B3 

mm 

EH 

EE 

bee 

^3B 

mnm 

^E 

bee 

ESH 

bsh 

be 

nu 

m m 

B-n 

nv» 

bsh 

IEH 

BSi 

BSI 

EHH 

ESI 

ee 

HBE 

EH 

IKE 

■B 

— 

wmm 

hlh 

bsh 

EH 

HE 

EE 

Till 

bsh 

bee 

ee 

KM 

hchI 

nOEEl 

mm 

Bi 

111 

a 

EE 

BE 

EE 

wmam 

KSH 

hee 

ee 

BE 

HEE 

mLm 

bm 

bee 

mm 

EE 

EE 

El 

EE 

mLm 

EE 

El 

H^S 

mu 

ESE 

HUB 

mm 

m 


EE 

EH 

wmzm 

EE 

HEH 

HH 

ESE 

BEE 

KM 

ESH 



mtm 

■[■ 

EE 

IEH 

hee 

mi 

MTPM 

bsh 

bee 

HBE 

BSI 

HBE 

HUE 

— 

wmm 

HI 

EE 

ES 

heh 

be 

KEEHB 

EE 

bqh 

HBE 

1BI 

BEBl 

hesib 

mm 

BS | 

IE£^I 


BE 

heh 

EH 

HTTOB 

mm 

ee 

IEH 

KM 

ESSE 

heih 

BBB 

Bsn 


KB 

BEI 

HEE 

EE 

helm 

EE 

BSH 

be 

bb 

ee| 

HE£fli 

mm 

BSH 

KLH 

E 

EE 

bee 

H3B 

heem 

KLH 

EM 

BESI 

HE 

be! 

^kee 

mm 

bsh 


EE 

RBI 

bee 

|EI 

mztim 

EH 

HBE 

BSH 

KM 

HBE 

HEUE 


BBS 

KLH 


■EH 

huh 

eh 

m mm 

KLH 

B 

e^h 

EH 

HEE 

KB 

HR 




EE 

HEE 

EH 

i»n 

EH 

HKH 

BSH 

^E 

hehI 
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TABLE 10. PROGRAMrONE, VERSIONS:l-13, REPLICATIONS:26-50 
(Version 8 did not fail on replications 2, 24, and 99.) 


REP 



mm 

BB 


Bl 

~n 

“5“1 

—n 

4 


6 

7 

8 

9 

BIBI 

UBI 


Kl| 

26 

Ui 

^■1 

Ka 

— 

— 

— 

— 

— 

— 

— 

— 

— 


27 




— 

— 

— 

— 

IKES I 

— 

— 

— 

— 


28 

-n 


mim 

— 

— 

— 

IBB 

EH2BI 

— 

— 

— 



29 

T 


mzm 

— 

— 

— 

EBI 

mam i 

— 

— 

— 



30 

mm 

wmm 

o 

— 

— 

— 


E£HB 

45 

— 

Bail 

mvrm 


31 


2 


— 

— 

— 

— 


— 

— 

— 



32 

T 



ia 

— 

— 

— 

— 

tua 

— 

— 

— 



33 

mm 



o 

— 

— 

— 

— 

I^BSI 

13 

id 


21 


34 


2 

ta 

. — 

— 

— 

— 

mmm 

— 

mzm 

“ 



35 

i 



ua 

— 

— 

23 

— 

msm 

— 

KB 

— 

— — 


36 

i 

2 

m 

— 

— 

— 

— 

m mm 

— 

mm 

— 



37 

i 






— 

— 

— 

mzm 

— 

— 

— 

m ib 


38 

i 




— 

— 

— 

— 

mmm 

— 

— 

— 

- 


39 

i 

2 

s™ 

— 

— 

— 

— 


— 

— 

— 

mmm 

UBI 

40 

i 



El 

— 

— 

— 

— 

- 

67 

— 

— 

hub 


41 

i 

2 

KOI 

— 

— 

19 

— 

KOI 

KB 

20 

— 



42 

i 

2 


M* 

30 

— 

— 

iua 

23 

— 



. 

43 

i 



■ra 



— 

mum 

iBB 


— 

64 




44 

i 

2 

r=- 

— 

— 


— 

IKS 

20 

— 

— 

80 


45 

i 


mm 

IKS 

— 

— 

— 

IKQUa 

5 

— 

— 

16 

_ 

46 

i 

5 


nr 

— 

— 

— 

isma 


— 

— 

■**“ 


47 

i 


10 

h=“ 

— 

11 

— 


IKQB 

53 

— 

— 

— 

48 

i 



ikeb 

IB 

— 

— 

— 

lama 

\mssm 

IBi 

— 

— 


49 

i 

7 

IK YM 


isa 

IB 

— 

\mma 

■Ui 

— 

■ — 


■ 

50 

i 

— 

IK£B 


i 

ce: 

— 

| 454 

r tt— 


L^=~ 
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ORIGINAL PAGE 13 

OE POOR QUALITY 


TABLE 11. PROGRAM:ONE, VERSIONS:l-13, REPLICATIONS*! :75 


REP 

■■■— i i ii i ■ i ■ 1 1 ii i ■ hni nil l r— ■ 


mm 



4 

5 

e 

7 


* 

mm 

mm 

mm 

13 

SI 

i 

KOI 

o 

— 

— 

— 

— 


ml m 

— 

mm 

— 

— 

52 

i 

— 

2 

— 

— 

— 

— 


— 

— 

— 

35 

75 

53 

— 

— 

7 

— 

— 

— 

— 

m 

— 

El 

— 

— 

— 

54 

mm 

— 

KOI 

3 

mm 

— 

— 

163 

— 

— 

— 

— 

— 

55 

i 

— 

KXB 

— 

— 

— 

— 

WMHM 

19 

— 

— 

20 

— 

56 

MM 

— 

- 

— 

— 

— 

— 

MZEM 

— 

— 

— 

rntm 

— 

57 


2 

mm 

— 

— 

— 

— 

- 

3 

— 

— 

66 

— 

58 

l 

3 

mm 

— 

— 

— 

— 

msm 

23 

— 

— 

79 

— 

59 

i 

4 


mm i 

MEM 

MEM 

— 

e urn 

— 

— 

— 

— 

— 

60 

mm 

— 

mm 

Km 

— 

— 

— 

mrnm 

— 

— 

— 

— 

— 

61 

i 

— 

mm 

. — 

— 

— 

— 

■IVil 

27 

— 

— 

48 

— 

62 

i 

— 

mm 

— 

— 

— 

— 


— 

— 

— 

— 

— 

63 

i 

— 

3 

mm 

mm 

mm 

— 

m 

— 

— 

— 

— 

— 

64 

i 

2 

— 

— 

— 

— 

MEM 

warm 

— 

— 

mm 

— 

— 

65 

i 

— 

2 

— 

— 

— 

— 

mnm 

31 

mm 

— 

— 

— 

66 

mm 

3 

mm 

— 

— 

— 

— 

mr\im 

84 


— 

— 

— 

67 


— 

8 

— 

— 

— 

— 

kmi 

mem 

— 

— 

mwm 

— 

68 

i 

— 

— 

— 

— 

— 

— 

TLTT'M 

warn 

— 

— 

Em 

— 

69 

i 

— 

■m 

— 

— 

— 

— 

KQJI 

— 

— 

— 

- 

mum, 

70 

■■ i 

— 

mm 

— 

— 

— 

— 

635 

— 

— 

— 

msm 

— 

71 

mm 

2 

mm 

„ 

— 

— 

— 

349 

— 

— 

— 

— 

— 

72 

i 

— 

9 

— 

— 

— 

— 

5 

— 

— 

— 

20 

— 

73 

nr 

2 

o 

— 

— 

— 

— 

mssm 

13 

— 

— 

17 

— 

74 

mm 

— 


— 

— 

— 

— 

mum 

El 

— 

— 

mm 

mum 

75 


2 


— 

— 

— 

— 


- 

— 

— 

35“ 
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TABLE 12. PROGRAMrONE, VERSIONS:l-13, 
REPLICATIONS:76:100 


REP 


wmtm 

mm 


i 


— 3 — 

4 

HE2H 


7 

8 

9 

10 

ii 

12 

19 

76 

i 

5 

6 

— 

— 

— 

— 

3118 

— 

— 

— 

81 

— 

77 

i 

— 

19 

— 

— 

— 

— 

— TTm 

— 

15 

— 

— 

— 

78 

i 

3 

31 

— 

— 

— 

— 

Ha 

— 

— 

— 

12- 

— 

79 

i 

— 

3 

— 

— 

— 

— 

Tim 

48 

— 

— 

■El 1 

— 

80 

2 

— 

7 

— 

— 

— 

— 

wmmm 

20 

— 

— 

Km 

T— 

81 

1 

2 

14 

— 

— 

— 

— 

763 

— 

— 

— 

— 

— 

62 

1 

— 

— 

mm 

26 

— 

— 

6205 

— 

— 

— 

— 

— 

83 

1 

5“ 

6 


— 

12 

— 

— 

20 

14 

— 

— 

— 

84 

1 

3 

27 

— 

— 

— 

— 

■mm 

37 

— 

— 

69 

— 

85 

1 

5 

9 

-i 

— 

— 

— 

3962 

— 

8 

— 

— 

— 

66 

1 

— 

— 

— 

— 

— 

— 

3951 

— 

41 

— 

— 

— 

87 

1 

— 

31 

— 

— 

— 

— 

1206 

— 

— 

— 

42 

12 

88 

1 



34 

— 

— 

— 

— 

4366 

4 

i ii 

— 

— 


89 

1 



2 

— 

— 

— 

— 

551 

— 

— 

— 

23 

216 

90 

1 

— 

— 

— 

— 

— 

— 

1113 

— 

— 

— 

— 

— 

91 

1 

— 

3 

— 

— 

— 

— 

360 

20 

_ 

— 

65 


92 

1 



— 

— 

— 

— 

— 

3368 

74 

— 

— 

— 

215 

93 

1 

— 

11 

__ 

21 

— 

— 

1104 

6 

— 

— 

— 

— 

94 

1 

4 

15 

— 

— | 

— - 

— • 

wtmsM 

99 

— 

— 

22 


95 

1 



— m 

— 

— 1 

— 

— 


i3 H 

— 

— 

31 

42 

96 

1 




— 

— 

— 

— ! 

2450 

21 

19 

— 

— 

— 

97 

1 

3 


— 

— 

— 

— 

1216 

58 

— 

— 

10 

— 

98 

1 

— 1 

■ ■ 

— 

— 

66 

— 

743 

25 

— 

— 

— 


99 

1 

2 

9 

— 

— 

— ' 

— 

9594 

— 

— 

— 

6 

■a 

100 

1 1 

2 

4 

— * 

— 

— 

■ESI 


■I 


MEM 


IHEOI 

K mm 


iff! 

MESSM 

mvi-m 

km 


mwm 

mzTivbm 

mmm 

g JEM 

KI2JI 


mum 
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ORIGINAL' PAGE IS 
OJE POOR QUALITY 


TABLE 13. PROGRAM:ONE, VERSIONS:14-26, REPLICATIONS: 1-25 





TABLE 14. PROGRAM:ONE, VERSIONS:14-26, REPLICATIONS:26-50 
(Version 17 only failed on replications 4, 7, 14, 25, 33, 52, 53, and 89.) 




mm 

HLUH 

gmg 

ggiO 

gQ3 

3Iifl 

3X3 

mm 

mm 

Bui 1 

K13 

mim 

K±a 

mm 


■SB 

gg3 

beo 

mmm 

3313 

BIBI 

BSB 

BB 

Till 

WtM 

mm 

BSB 

BBI 

HE2BR 

BIBi 

gggg 

g33 


3113 

BEO 

BOI 

KB 

BBSB 

BSSB 

mm 

B^fl 

BSB 


o 

gggp 

obi 

mam 

mm 

bulb 

BSB 

BIB 

BBSB 

BSB 

B1 

BSB 

BBI 

HEUH 


BBS 

gg 

go 

3B3 

msm 

kb 

BBI 

BIBB 

BEB 

KJB 

EB 

BB 

IK23H 

BBSS 

O 

3333 

30 

msm 

o 

bb 

BSB 

OB 

OB 

BB 

BB 

EB 

ggjO 

o 

O 

3323 

msm 

«*■ 

OB 

KB 

BSB 

m 

kb 

mm 

BB 

BSB 

HEBi 

g| 

3££B 

3303 


go 

fW 1 B 


BB 

BBSB 

BUB 

BB 

K£B 

IBB 

HESH 

O 

gcg 

gggO 

KH£B 

HZ9 

OB 

09 

BSB 

K2B 

^B 

EB 

BSB 

BB 

mim 

3 

331 

wbsm 

mu 

mm 

bob 

o 

E3 

BBSB 

BSB 

BSB 

BB 

^B 

ggxo 


3331 

BB|B 

KOI 

OB 

i^BB 

OB 

mm 

BIBB 

BSB 

BSB 

OB 

BSB 


31 

ggi 


KJ« 

3X3 

BEB 

OBI 

kb 

BBSB 

BSSB 

BSB 

BB 

IBB 

BEDH 

g|OB 

gg3g 


KSE1 

3Q3 

B2EB 

BB 

BSB 

B^Si 

KB 

BB 

BB 

EB 

HE2B1 

gg^j| 

K£fi 

3BI 

msm 

bob 

IOB 

BSB 

BB 

■BEB 

3KB 

KB 

BUB 

OB 

3333 

^g 

gg 

09 

■VCTM 

ggOI 

iELB 

BSB 

BSB 

ggOB 

kb 

BSB 

BSSB 

BSSB 

IKI3B 

gg 

BELB 


1QEEB 

ggo 

| OB 

BBI 

^O 

miiim 

BB 

BB 

BIB 

EB 

HD3HI 

HESB 

HE3I 

3323 

EM3B 

o 

OSB 

BIO 

BSB 

BBEB 

BO 

BSB 

BSB 

BEB 

OQB 

mm 

mm 

gg 

■ila 

330 

ggo 

BSB 

o 

B B 

BB 

KB 

BB 

BSB 

ggggg 


31 

BEXBB 


msm 

OBB 

geo 

kb 

B^B 

IBB 

BSB 

IOB 

B^B 

B 2 BB 

31 

333 

gggQg 


ggo 

BEB 

BSB 

BEB 

bb 

BSB 

IBB 

OB 

BSB 

BbSH 

IBB 

bob 

wmm 

32mJJg 

BO 

BSB 

BlUi 

BB 

BB£B 

BBI 

BSB 

BSB 

BSB 

HKOH 

gg 


g^g 

gEESB 


IOB 

BOI 

OB 

kibi 

BSB 

BO 

BKB 

BSSB 

iKQHI 

31 

31 

31 

■ W»f 

gg 

ggo 

o 

OSB 

ggiSB 

IOB 

BBI 

KB 

1EB 

HCBI 

BOB 

ggo 

BBEB 

mmm 

KB 

BEB 

KB 

BB 

B^SB 


BBi 

BB 

BB 

KLB 

osb 

313 

keek 


beo 

o 

BSB 

IBB 

BBSB 

BSSB 

B 


BE 

m 

BEB 

BBi 



BB2B 

g mm 




BUB 
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ORIGINAL PAGE IS 
OF POOR QUALITY 


TABLE 15. PROGRAM:ONE, VERSIONS: 14-26, 
REPLICATIONS:51-75 
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TABLE 16. PROGRAMrONE, VERSIONS.-14-26, 
REPLICATIONS:76-100 


REP 



~~n 

13 

— TS 

HKtflH 

16 

19 

mm 

mam 

22 

23 

KXJH 

Bl 

kb 

76 

— 

— 

— 

6471 

120 

167 

— 

— 

— 

38 

— 

— 

— 

77 

— 

— 

279 

— 

KJi 

— 

— 

9 

— 

— 

— 

— 

— 

78 

— 

63 

262 

■KEE9 

- 

— 

— 

— 

— 

2 

— 

— 

— 

79 

— 

— 

— 

H2£DB 

KOI 

— 

— 

— 

— 

— 

— 

— 

— 

80 

— 

46 

224 

7167 

— 

— 

— 

— 

— 

— 

— 

— 

— 

81 

— 

— 

658 

■BE&H 

390 

— 

— 

— 

— 

33 

it 

— 

— 

82 

— 

37 

231 


— 

— 

— 

— 

■ea 

— 

— 

— 

— 

83 

— 

— 

■KES 

vga 

— 

— 

— 

— 

mmm 

— 

— 

— 

— 

84 

— 

— 

HB9 

3586 

203 

WEJM 

— 

— 

— 

— 

— 

— 

— 

85 

— 

— 

216 


ua 

— 

— 

3 

— 

— 

— 

— 

— 

86 

— 

— 

— 


30 

— 

— 

mtm 

— 

mBm 

— 

— 

— 

87 

— 

— 

— 

8651 

— 

40 

— 

— 

— 

mm 

— 

— 

— 

88 

— 

— 

28 

5320 

78 

— 

— 

— 

— 

- 

— 

— 

— 

89 

— 

— 

— 


— 

289 

— 

— 

— 

HI 

— 

— 

— 

90 

— 

— 

— 

■BgB 

77 

71 

— 

— 

— 

mtm 

8 

— 

— 

91 

— 

HEXS 

— 

9422 

— 

— 

— 

— 

— 

— 

— 

— 

— 

92 

— 


— 

6181 

— 

123 

— 

— 

— 

— 

— 

— 

— 

93 

— 

424 

209 

8232 

— 

— 

— 

— 

— 

— 

— 

— 

— 

94 

— 

25 

1011 

6668 

— 

— 

— 

— 

— 

— 




95 

— 

— 

— 

6513 

— 

24 

mm 

— 

— 

— 

— 

— 

— 

96 



— 

266 

7114 

msm 

— 

— 

— 

— 

— 

— 

— 

— 

97 

— 

— 

1305 

7236 

bqi 

— 

— 

— 

— 

— 

— 

— 

— 

98 

— 

— 

272 1 

8898 “ 

- 

— 

2 

— 

— 

■Bl 

— 

— 

— 

99 

— 

— 

114 

— 

Ka 

— 

— 

— 

— 

■a 

— 

— 

— 

HEEZ3H 

— 

— 

— 

8827 

. - 

73 

— 

— 

— 

* 

— 

IBB 


IHgjM 


muum 



■ Wf 

4859 

12 

3*3 

4345 | 

520 

352 

mm 

in 
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ORIGINAL PAGE IS 
OF POOR QUALITY 


TABLE 17. PROGRAM:ONE, VERSIONS:27-39, REPLICATIONS: 1-25 


REP 

iMiiiiiwiMBi 1 1 1 in Mil i Tirrm a ■ i nn ebe—bbeb 


mm 

28 

mm 

mm 

mm 

mm 

mm 

wtm 

mm 

mm 

wtm 

mm 

mm 

1 


— 



■SB 

S3 

BS9 

wssm 

msm 

B3I 

ESI 

msm 

mo 

2 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

3 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

4 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

5 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

6 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

7 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

8 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

9 

— 

— 

— 


B91 

BSK 

msm 

msm 

mom 

— 

— 

— 

— 

10 

— 

— 

— 

- 

- 

- 

- 

- 

- 

— 

— 

— 

— 

11 

— 

— 

KS 

Ball 

31 

IBB 

mo 

msm 

WBO 

mo 

BE 

— 

— 

12 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

13 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

14 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

15 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

16 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

17 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

18 

m m 

wmcsm 

E5H 

USB 

msm 

msm 

■e 

BBS 

msm 

ebb 

msm 

li*** 


19 


— 

— 

— 

— 

— 

— 

BSi 

mo 

mo 

BBS! 

ESI 

BOB 

20 

— 

■*H>M 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 


21 

— 

HBSSB 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 


22 

— 



m 

mo 

msm 

BBS 


mm 


mom 

MSM 

■aii 

23 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 


24 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 


25 

— 

— 

— 

— 

— 

— 

msm 

HOI 

— 

— 

— 

— 

gga 
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TABLE 18. PROGRAM:ONE, VERSIONS:27-39, REPLICATIONS:25-50 
(Version 33 failed on Replication 53 only.) 



ORTTMAL' IS 

OF. POOR QUALITY 


TABLE 19. PROGRAMrONE, VERSIONS:27-39, 
REPLICATIONS:51-75 
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TABLE 20. PROGRAM:ONE, VERSIONS :27-39, 
REPLICATIONS:76-100 


REP 


ua 

BB 

B 

■ B ' 


mim 

55 

“55“ 


til 

B±» 

B&JHI 

mum 

tai 

Kill 

mxm\ 

IMI 

ElMj 

76 

m 



— 

— 

Bail 

1=1 

■an 

— 

— 

— 

. 

— 

— 

77 

— 

— 

— 

— 

— 

■aw 

mrcrm\ 

— 



-E“ 


-EE 

78 

79 

-EE 







— 

— 

— 

— 

— 

— 


— 

— 

80 

81 

— 

— 

— 






— E- 

— 

— 

— 

-EE 

— 

— 

82 

83 

— 

— 

— 


— 

— 

— 

— 

— 

- 

“IT— 

— 

— 

84 

85 


— E— 




__ 

— 

— 

— 

— 

— 


— 

— 

86 

1 



— 

— 

— 

— 

— 

— 

— 

— 


*— 


87 

88 
89 

— 

— 

— 

- 

- 

- 

— 

- 

- 

- 





90 

91 

92 


- 

— 

— 

- 




— 



-=T 

2 



93 

94 

— 

— 

— 



— 

“ 

— — 

— 

— 

— 

-E“ 

— 

a 

95 

96 

— 

— 

— 

— 

— 



— EE- 



— 

— 

-EE 

— 

m 

a 

97 

98 

— 

— 





— 

— 

— 

— 

— 

— 

E •• 

— 

m 

99 

100 

— 

— 

— 

— E- 

-EE- 



iK=a 

IBB 

IK 

m 



IBB 

imam 

\mm 


iiiga 

iia 

i mm 

\mm 

M&M 

252 

■a 

\mm 

IKjJ 

1 ss 

i_MLJ 



ORIGINAL PAGE 15 
OJE POOR QUALITY 


TABLE 21. PROGRAMrONE, VERSIONS :40-45, REPLICATIONS:1-50 
(Versions 42, 43, 44, and 45 never failed.) 





TABLE 22. PROGRAM:ONE, VERSIONS:40-45, 
REPLICATIONS:51,100 


ORIGINAL' PAGE IS 
OE POOR QUALITY 


TABLE 23. PROGRAM:THREE, VERSIONS:l-13, 
REPLICATIONS:l-25 
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TABLE 24. PROGRAM:THREE, VERSI0NS:1-13, 
REPLICATIONS:26-50 

(Version 7 did not fail on Replications 5, 24, 56, and 75.) 


■ran 


■Hg 

MM 

BfcH 

KJH 

bh 

BIH 

bh 

bh 

bjb 

blb 

hub 

HIM 

HLH 

MSM 


n 

MEM 

msm 

BEH 

BEH 

bbsb 

bbsb 

HSI 

bh 

BiH 

HE3B 

HBH 

KH 

HEQH 

KB 

MEM 


HIM 

bh 

BK3 

blsh 

BEH 

bbsb 

BBSB 

HH 

BEEH 

^3B 

B£B 


^31 

msm 

KH 

bh 

BflUXH 

MMM 

BEH 

KH 

HI 

■KH 

BEH 

EH 

MELH 

MM 

MxM 

B^B 


bh 

■FTYM 

HK^B 

HH 

HE 

HI^H 

BB^H 

KH 

^3B 

IKI3I 

w 

EH 

B 

MMM 

bh 

HOB 

b*mm 

EBB 

b^lh 

BEH 

BHB 

^BH 

^H 

BEQH 

BE 


MSM 

BBHI 

IBB 

hbeh 

bh 

BSB 

H^H 

BHB 

B±£B 

EM 

BEH 

B5QH 

MM 

mm 

msm 

MIM 

bh 

HE£H 

mmm 

bh 

BH 

BEH 

BBHI 

BH 

BBH 

MEEBB 

WEM 

mm 

EH 


■be 

BH 

msm 

MSM 

KIM 

BEH 

BEB 

B^H 

BOB 

HEDH 

MM 


msm 

bh 

bh 

BEH 

BQQB 

eh 

EELOfl 

BHB 

HBH 

HH 

KH 

IKQH 

MM 

HIM 

BEH 

bh 

HH 

■BEH 

beh 

ESH 

beh 

msm 

EH 

MEEM 

bh 

BXMI 



EE 

HH 

bh 

BEH 

beh 

bh 

BE 

HEM 

HE£H 

HB 

HSB 

HEQH 

MM 

MEM 

B3B 

EH 

bh 

B9 

hbh 

^H 

bh 

BEH 

beh 

Hi 

^^H 

IEOH 

MM 

EH 

MSM 


bhb 

Bfl^H 

beh 

boh 

HIM 

BH 

BH 

KB 

^H 

HEQH 



BEH 

KH 

BQH 

BH 

■beh 

IB 

BEH 

BEH 

BBSB 

BH 

^H 

mxm 

MM 

MtiM 



bh 

BE2£H 

beh 

beh 

MTFM 

BH 

hbh 

HEH 

BH 

HDHI 

MM 

EH 

Bl 

MEM 

hxh 

BEH 

■SZH 

eh 

bh 

BBSB 

BB 

BH 

BSB 

flESH 

MM 

KH 

eh 



bkem 

beh 

MEM 

beh 

BH 

BH 

HBH 

BBH 

E3^B 

MM 


bee 

BEH 

mm 

bsb 

bsb 

HH 

BEH 

BEH 

HIHI 

BH 

BBH 

EE0H 

mm 

mm 

bsb 

HSB 

BEH 

bh 

be 

bh 

bhb 

MMM 

BBUB 

BH 

eh 

HCEHi 

MM 


BEH 

bh 

n 

beh 

bbsb 

beh 

HBH 

BBSB 

MHSB 

HSB 

^SM 


MM 

i mm 

BSB 

bh 

bh 

MMI«M 

IBEH 

MIM 

beh 

BBHI 

■BHB 

HH 

IHI 


BE 

HJB 

BE 

mm 

BBH 

beh 

bqhi 

BEH 

BH 

HBH 

HBH 

BBSB 

HH 



O 

B9B 

MEM 

BEH 

HEQH 

beeeh 

bh 

bh 

IBEH 

HBH 

HSI 

BSB 

IEOH 

MM 


bh 


bh 


BH 

■HI 

be 

beh 

BE 

BEH 

BBH 

IMQjHB 

BE 

MSM 

KH 

HH 

EH 

EflESH 

■ni 

KH 


HESS 

HEBH 

H 

KH 
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ORIGINAL PAGE IS 
OJE EOOR QUALITY, 


TABLE 25. PROGRAMrTHREE, VERSIONS:l-13, 
REPLICATIONS:51-75 




H^B 

mm 

KB 

KB 

BB 

BEB 

BUB 

K BBB 

bub 

BUB 

KJB 

IKUE 

BIB 

KM 



BOI 

kb 


BSB 

■T*B 


bsb 

BCB 

■SB 

HESS 

BSS 

^B 


KB 

BIB 

BCB 


rn^vm 



KB 

b^b 

BEB 

■1 

^EB 

BBS 

ggi 



KS 


m 

BB£B 

BI^B 

EB 

BEB 

fl^B 

■B 

■s 

^B 


— 

BB 


BSB 

KK 

bbb 

bbb 

BB 

m 

MM’g 

KliB 

Bi 

EB 

■E£H 

mm 

BIB 

■ 

BSB 

mi 

BBB 

bibb 

bb 

BEB 

BOB 

B£3 

BT&TB 

BSB 

■EUB 1 

— 


■ 

mu 

BB 

IK^B 


BB 

beb 

EES 

BEB 

BESS 

BB 

m 

mm 

IBB 

■ 

keb 


KEElB 

■TEEM 

EB 

■3 

EBB 

EES 

■EB 

BB 


mm 

BIB 

■ 

BUB 

EH 

mum 


■ 

beb 

EES 

B 

kbss 


HEEHE 

mm 

BSB 


K£B 


kub 

bbbi 

BSB 

I^BI 

BBB 

BBB 

EBBB 

B3B 

BE3H 

KB 






bbb 

bsb 

BEB 

m 

bob 

BEB 

^^B 

HBHI 

KB 

bib 

KS 

BSB 

B 

■US 

gw 

kb 

K LIB 

BESS 

BBB 

^B9B 

i^B 

HQBi 

KB 

bu 

■ 

BSB 

BB 


kbb 

EB 

bbb 

BCS 

«« 

BBSS 

n 

HCSH 



■ 

mm 

kb 

ETT'B 

bbb 

EB 

K£B 

B^B 

BEB 

BSI 

i^S 

BQ3H 

f 

BIB 

■ 


BB 

bub 

^bbi 

kb 

beb 

BEB 

^BBB 

■ESS 

EB 


KB 


BIB 


ELB 

■Ifl 

bhub 

bsb 

BEB 

BIBB 

BEB 

BSB 

BS 

HSU 


mfm 


keb 


■ffl 

i^KSSI 

mi 

B^B 

EBB 

BBESB 

IBB 

IB 

KQB 

s 

mm 


bsb 

BB 

B 


EB 


EUB 

cia 

jjjgjISB 

B 

■EB 

mm 

mm 

WSM 

beb 

BB 

BB 

■ B 

^B 

Wf*B 

mB 

BBSS 

SB^SB 


KQEI 

KB 

BJI 

B 

BSB 

BSB 

■BB 

BEB 

KSB 

BEB 

BBS 

KLHS 

j^EB 

dbJI 

BKCH 


mwm 

IBB 

: fflB 

IBB 

iKmi 

B^BI 

ksb 

BESS 

BB 

■££B 

EB3B 

I^S 

BB^B 

KB 


KB 

BB 

KB 

Bm 

BM&B 

bb 

beb 

IBB 

BBS 

■ 


BSBI 


IBB 


ksb 


bub 

KB 

bsb 

BSB 

BEB 

EBBB 

BBSS 

ESS 

BB^B 

KB 

jKB 


m&m 

beb 

■gyyB 

BBBI 

mm 

BEB 

^BB 

SEB 

BBB 

IBS 

■BBI 


BB 


k m 

IBB 

bob 

■:HrM 

BBS 

E^B 

BBS 

■ESS 

BBB 

BSS 

Bn 

\mm 




BB 

■IHJ 



BIB 


S^ 
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TABLE 26. PROGRAMtTHREE, VERSIONS:l-13, 
REPLICATIONS:76-100 




BXXB 

^B 

mm 

IHflH 

XEHI 

gj| 

hlh 

BJX 

buh 

bh 

BJX 

BXfl 

buh 

BKIXB 

HiJX 

BIX 


BOg 

HEiX 



gB 

bhjx 

BBHSX 

XEHI 

gXLLX 

BXH 

XBEH 

HEH 

BEX 


IHftfl 

IXE^X 

B 

XX^H 

bh 

B H 

BS 

BS 

BO 

bh 

XBLUX 

HEH 

IKB 

Hzm 

gBX 

BHHI 

BHB 

BIX 

BEX 

HII1X 

bkhx 

XEX 

XBH 

BKH 

XBEH 

KS 

E^X 

BXJX 

BEB 

BX 

>x 

g|BX 



bh 

bh 

■^n 

BXH 

HBEX 

Kl 

KH 


BX 

I^MSX 

gg 

XEIX 

bh 

■R« 

XXHX 

IBS 

BH 

BXEX 

XBBSX 

HBHI 

i^^E 

BOflP 

I9KX 

EIKX 

KH 

BIX 

HEX 

BEX 

XX~~"X 

S^HI 

BEH 

XBSX 

BXEX 

HEH 

BOB 

IB2H 

mM 


Bi 

HEX 

HEX 

BBBUX 

bxeux 

■eh 

IHi£X 

BKH 

HIHB 

i^H 

KH 

B2BI 

HEX 

XXDB 


XEHI 

XEH 

bbiex 

gX£3 

H 

HEiX 

BBSS 

X^HE 

HEH 


flEZIH 

HKB 

XHLX 

IBB 

BEX 

XE9B 

gBH 

XKH2X 

Bl 

BB1X 

BIBB 

BBKEB 

IBEX 

XEX 

BJXI 

m 

XXEX 

gg 

I^EX 

Bl 

XHQX 

XEH 

EXHI 

XBSX 

BKH 

XEX 

BEE 

KX 

gBSB 

HKX 

BX 

B 

IKH 

HHB 

BKX 

BBS 

BEH 

EXiX 

BKH 

fli^EB 

HEX 

E^X 


mm 

■BIX 

KH 

HEX 

BEX 

XB22X 

BS 

BIBB 

^XBH 

HEH 

XHX 

I^EHI 

ESH 

B^JBI 

HHX 

^XBX 

EBX 

B 


bhx 

BBBQEB 

BBSS 

XX£X 

BKH 

BXEH 

HX 

KH 

MB3H 

HEX 

■BX 

BHI 

BX 

BIX 

bekx 

BXE2X 

■BOB 

XBHI 

^KSX 

IHE 

BH 

KX 

lEEH 

IXEB 

HKX 

gg 

B 


eblix 

bkx 

i^cx 

XEHI 

HEX 

BHEH 

BXEX 

XEX 

bzb 

^BX 

HKX 

B 

HSX 

XEH 

BHBX 

BIHEH 

■BS 

BH 

HEX 

Bfl 

HEH 

B5X 

IKEHI 

HEX 

B 

ggg 

B 

PS^SI 

sox 

BBS 

BEH 

■HX 

xs 

XB 

BHa 

BEX 

HBH 

HEX 

XHX 

gg 

B 

XEX 

BESX 

bh 

BIX 

heh 

BXEX 

XBBEX 

XXHI 

KH 

flEXHI 

BX 

HEIX 

IBX 


HSX 

giBH 

BBKBES 

XEH 

XHX 

XBEH 

HKiX 

EBBB 

XEX 

HEQH 

XBH 

HEIX 

X^X 

B 

,^BX 

BCJ2X 

bh 

BEX 

XBSX 

HEX 

BH 

XEH 

KH 

KZ3H 

XXX 

flflHX 

XEX 

X| 

B 

bh 

^BUX 

HHB 

ebesb 

BKH 

BBH 

XS 

BUH 

1K21H1 

BS 

BBBELX 

gg 

BOX 

B 

HUB 

XKH 

■eh 

bh 

■B H 

BKH 

,i^^H 

BH 

1KZ3K 

IHEX 

Ihex 

K 

BEX 

B 

BU2X 

BOSS 

IXJ 

BtLX 

BKH 

XBH 


KH 

HG3B 

XBH 

BKH 


IBS 

mZM' 

beux 

bxlux 

bh 

BKH 

BEX 

BXE3B 

XBH 

KdHi 

blxh 

^HX 

HEIX 

K 

iBEX 


ibqx 

XKU£X 

iXEIX 

HH 

BH 

iXBEH 

iXEX 

BEX 

mam 

SEX 

XESEX 

K1 

■201 

mtm 

Baai 

BB 

\mxm 

IBSX 

BffM 


luax 

XJjX 
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.ORIGINAL PAGE IS 
OE POOR QUALITY 


TABLE 27. PROGRAM.THREE, VERSIONS:14-26, 
REPLICATIONS: 1-25 




BE 

BE 

mm 

ai 

KIR 

HH 

Ell 

■LEI 

El 

El 

■Ml 

phi 

m 

El 

EEI 

m 

K1B 

EB 

mm 

EBI 

■B: 

jg 

■Mi 

^Bl 

^E: 

mm 

kei 

an 

^Ri 

I^EBI 

El 

El 

PSPBi 









mm 

gg 

■M 

■eeb 


ggg 

EB 




s 


B 

s 

s 




&B3J1 

B 

EB 

■31 

BEB 

■SB 

EB 

E 

IEB 

E 

EB' 




E 

31 

■SB 


EE 

l^B 

EB 

E 



Epii 

IKQ^B 





31 

E 

BEB 


EB 

EB 



n*»R 


gg 



E'ER 

EB 

E 

■EB 

EB 

EB 

EISE 


EB 

B 

gig 



MEB 

If^B 

ksb 

eb 

BEEZB 

■SB 

EB 

E 

IK® 







3^^3 

■SB 

BE 

wmm 

KLB 


E 

Ell 

«s*=B 



ESB 

£3 


^KR 

EB 

BEB 

BE 

■SB 

EB 

B 

EB 

EB 

EB 


Eli 




31 


■ EMB 


E 

IE 

iEB 



SrS 

EM 

■ 

■ 





j^g 

g 

mm 

g 

g 

g 

g^g 


gB| 

g 

g 

g 

g 


ra 


IB 

j^B 


EB 

£5 

;E 



BH 

32 


31 

mm 

IEB 

BIB 

EB 

IEB 

■SB 


IEB 


wra 



E 

■B^l 

IEB 

\msm 

IEB 

U| 

IBB 

■KB 


a 



E 

3 


BEB 


IE 

IEB 


IEB 

J3B 


a 



E 

E 

E3 

■BIB 

IE 

I EH 

IEB 

E3 

IEB 

Er!*j 


IBB 

M 


E 

E 

EB 

B'lER 

!E^ 

IEB 

IEB 

E-^ 

IE 

EB 

ikhi 

IEH 



E 

E 

E**f 

l^Efl 

IE 

IIE 

IBB 


iEm 




■ 

■ 

■ 

ESi 

E 

■HER 

iBgjj 

1*55 

LE 

E£| 
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TABLE 28. PROGRAMrTHREE, VERSIONS: 14-26, 
REPLICATIONS:26-50 





ORIGINAL PAGE IS 
OF POOR QUALITY 


TABLE 29. PROGRAM.THREE, VERSIONS: 14-26, 
REPLICATIONS:51-75 


REP 

■hrrh— ri i;i*h ) u— i — i 


U1M 

mm 


17 

mm 

mm 

20 

mim 

mm 


\mim 

mm 

mm 

51 

— 

— 

— 

26 

— 

— 

— 

- 

- 

mm 

- 



52 

— 

— 

— 

— 

— 

— 

— 


ESI 

bsh 

^SM 

HSI 

HStf 

53 

— 

— 

— 

Tim 

— 

— 

— 

2 

mm 

E 

— 

— 

— 

54 

— 

— 

— 

— 

— 

— 

HI 

— 

— 

— 

— 

— 

— 

55 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

56 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

57 

— 

— 

— 

98 

— 

— 

— 

— 

— 

— 

— 

— 

— 

58 

— 

— 

419 

— 

— 

— 

— 

— 

— 

— 

— 

— 

— 

59 

— 

— 

73. 

KOI 

— 

— 

— 

Si 

B3R 

a 

R3R 

KB 

a 

80 

— 

— 

— 

— 

— 

— 

mwm 

— 

mm 

HSR 

HSR 

SR 

^SR 

61 

— 

— 

— 

— 

— 

— 

- 

— 

— 

— 

— 

— 

— 

62 

— 

— 

— 

— 

— 

— 

mum 

— 

— 

— 

— 

— 

— 

63 

— 

— 

976 

u 

— 

— 

— 

— 

— 

— 

— 

— 

— 

64 

— 

— 

580 

— 

— 

— 

— 

BIS 

B3I 

SSI 

31 

eh 

msM\ 

65 

— 

— 

— 

— 

— 

— 

— 

- 

- 

- 

- 

- 


66 

— 

— 

633 

16 

— 

9 

— 

BS9 

rssi 

rqr 


RSRI 

BI 

67 

— 

— 

— 

— 

— 

— 

EXLB 

— 

— 

— 

— 

— 

— 

68 . 

— 

— 

86 

— 

— 

— 

- 

— 


— 

— 

— 

— 

69 

— 

i — 

— 

— 

! — 

UBS 

BSESR 

— 

2 

— 

— 

— 

— 

70 

— 

— 

— 

— 

— 

Bi 

mwm 

— 


— 

— 

— 

— 

71 

— 

— 

— 

— 

— - 

— 

— 

— 


— 

— 

— 

— 

72 

— 

mH: 

KE9I 

— 

— 

— 

— 

6 

— 

— 

— 

— 

— 

73 

1 — 

■Sili 

■KIM 

— 

— 

— 

— 

— 

msm 

BSS 

RSR 

KSH 

BSi] 

74 

— 

— 

— 

51 

— 

— 

— 

— 

— 

— 

— 

— 

— 

75 

— 

— 

— 

35 

*— 

— 

— 

mm 

— 

— 

— 

— 

— 
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TABLE 30. PROGRAMrTHREE, VERSIONS: 14-26, 
REPLICATIONS:76-100 


BMS 


■■ 

BIB 

mm 

bbb 

BBUB 

BIB 

BIB 

IBKlIBB 

Bifl 

mm 

Bl£B 

B IB 

BIB 

mm 





BBSB 

BSB 

BBB 

bbsb 

bjb 

■a 

bb 

EB 

^SB 

i^B 

HK£H 

B3B 

BB 

bbsb 

bbsb 

BSB 

^58 

BBxLB 

bib 

EB 

BBSB 

BSB 

BE 

BSE 


gi 

EEE 

bbsb 

bbbi 

bb 

bb 

BBBSB 

BjESi 

BBB 

B 

BBS 

BE 

BSE 


B3I 

bb 

BBBSB 

^beb 

i^B 

BSB 

bbsb 

BBSS 

BB 

B^B 

■SB 

■Si 

EE 

HEH1H 

E 

B 

lhib 

bob 

BSB 

IBB 

BBBSB 

BES3B 

BEB 

BCE 

EE 

EB 

i^E 

BEIHi 

EEI 

B 


bbb 

BSB 

BSB 

IBSSB 

bb^b 

BBSB 

EE 

BSE 

IBB 

BSB 

EZBB 


abb 


BKIB 

BSB 

IBB 

B 

Efl 

BSB 

BBSB 

BE 

BSE 

■SB 

HSH 


bbb 

B 

bbssb 

BSI 

IBB 

bibb 

BB B 

BSSB 

BSSB 

BSB 

BSB 

EE 

HEZSi 

B 

B 

bbsb 

bblb 

BSB 

BSSB 

BBBSB 

E8 

BBB 

BBSB 

EE 

BSB 

EB 

BESH 

ai 

BBB 

bbsb 

BKDB 

bb 

BSSB 

BBBSB 

BBB 

EB 

BB 

EB 

EB 

BSI 

HQ3H 

B 

IB3B 

Bmi 

B B 

bb 

BSB 

bbssb 

bb 

BEB 

BUB 

EE 

I^S 

BSE 

IKQHi 

IBB 


bub 

mBQAI 

BSB 

BSB 

BBBSB 

BE 

BEB 

BSB 

BSB 

ABIE 

EE 

BQ3H 

BSB 

IBB 

B 

B 

BSB 

bb 

BBEB 

bbb 

BESS 

BSSB 

B^B 

i&SSB 

BSI 

IKEHI 

B 

BEBI 

bbsb 

B 

BSI 

BSSB 

BBSS 

BBSB 

BEE 

BBB 

BSI 

EB 

I^S 

fldB 

B 

B 

bbeb 

bllb 

BSB 

BBI 

BBSS 

■llb 

bbb 

BBSB 

IBB 

EE 

EE 

HQ9HI 

B 

B 

ebeb 

B 

BSB 

BSB 

bbub 

BSSB 

BEB 

BBS 

ESB 

BSE 

EE 

HiZIH 

IBB 

B 

BH 

bbeeb 

BS 

B2B 

BBEB 

■BSB 

BEB 

BCE 

EB 

BSSB 

ESB 

HOBi 

BSB 

BSB 

bbub 

BBSB 

bb 

BSB 

bbes 

be 

BBSB 

BBSB 

EE 


EB 

■OB 

BB 

BBB 

BBSB 

BB 

m 

BSB 

beirb 

Bll 

■ 

53 

ggjjjg 


SB 

\mrrmm 

fcsS 

B 

bSbI 

bS 

fed 

151 

ss 

msm 

B-a 

Be 


o 

ee 

BCBB 

BSB 

B 


bbb 

BSB 

mm 

BBBSSB 

bbb 

BSB 

■SB 

■^E 

EE 

BSB 

'BEHB 

B 

BSB 

IB 

BBEB 

BSB 

IBB 

■bbssb 

bli 

eb 

ee 

BSSB 

IBB 

ESB 

BEEBI 

IBB 

gp£B 

bb?^p 

bbeb 

BSB 

BSB 

bbeb 

beub 

BEE 

BOB 

EB 

BSE 

EE 

BQH31 

EE 

BSB 

BBBSB 

bbsb 

BSSB 

BE 

BO 

BBB 

BSSB 

i^SB 

IBB 

EE 

B 

Baaai 


EES 

bsbb 


BBB 

mm 


IQB 

mum 


ELE 

mm 

BIB 



TABLE 31. PROGRAMrTHREE, VERSIONS:27-36, 
REPLICATIONS:1-50 

(Version 27 failed on Replications 20, 33, 71, and 84 only, 
Version 28 failed on Replication 25 only, and 
Versions 35 and 36 did not fail.) 




mm 

hob 


mm 

hub 

ftifl 

■<v 


BIB 

HUB 

hob 


b mm 


bsb 






BBSB 

hesb 

■H1H 

EM 

Hff^H 

BSfl 






BBSB 

BBSB 

■ran 

IB^B 

msm 







BEB 

BEB 

ns 

k mm 

bbsb 

ESI 

BBB 

BB 

EB 

BSB 

hsb 

BBSB 

BIBI 



BBSB 

beb 

BEB 

IBB 

BSB 

ESI 


BBSB 

BBSB 

HH 

■TOM 

H 9 

BSB 






BH 

BEB 

m 

■Mi 

H 

BSB 

beb 

BSB 

BSfl 

BSB 

^a 

BIBB 

bbsb 

■m 

bob 

B B 

bsb 

SE3 

BB 

BBI 

BSB 

hb 

BHB 

hhsb 

H'Sl 

mssm 

BBESB 
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